2024-02-19T21:58:56.378 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:58:56.378 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --archive /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809 --verbose --name yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/orig.config.yaml 2024-02-19T21:58:56.428 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:58:56.519 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809 branch: pacific-release description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566809' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9538 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi045.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIp0R+lKPi5Uh/zVH3WN/rPCPMWSYYk6gpVFwEE75Xpm+b2x4NQfSsvXV5SrA97Wg6YP70DCqbujCwosICzA5ME= smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmcWGiaGEhl8s/CNIY8wfTWhO4vF2x9vwAaa5jojaK/jtOXiUwr1XR+2PDzNafYJgYVgu1nKXq2ouSD3RY3SBI= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:27:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:58:56.520 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:58:56.520 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:58:56.520 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:58:56.522 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:58:56.545 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:58:56.546 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:58:56.546 INFO:teuthology.packaging:ref: None 2024-02-19T21:58:56.546 INFO:teuthology.packaging:tag: None 2024-02-19T21:58:56.546 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:58:56.546 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:58:56.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:58:56.691 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:58:56.693 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:58:56.699 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:58:56.699 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:58:56.733 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:58:56.751 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:58:56.757 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:58:56.782 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:53:48.744013', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIp0R+lKPi5Uh/zVH3WN/rPCPMWSYYk6gpVFwEE75Xpm+b2x4NQfSsvXV5SrA97Wg6YP70DCqbujCwosICzA5ME='} 2024-02-19T21:58:56.803 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:53:48.745429', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmcWGiaGEhl8s/CNIY8wfTWhO4vF2x9vwAaa5jojaK/jtOXiUwr1XR+2PDzNafYJgYVgu1nKXq2ouSD3RY3SBI='} 2024-02-19T21:58:56.803 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:58:56.810 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:58:56.810 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:58:56.810 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:58:56.886 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f99e1efef70>, signals=[15]) 2024-02-19T21:58:56.887 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:58:56.894 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:58:56.894 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi045.front.sepia.ceph.com 2024-02-19T21:58:56.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:58:56.971 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-02-19T21:58:56.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:58:57.047 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:58:57.054 DEBUG:teuthology.orchestra.run.smithi045:> uname -m 2024-02-19T21:58:57.077 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64 2024-02-19T21:58:57.078 DEBUG:teuthology.orchestra.run.smithi045:> cat /etc/os-release 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream" 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8" 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos" 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8" 2024-02-19T21:58:57.139 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:58:57.140 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:58:57.141 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server 2024-02-19T21:58:57.165 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-02-19T21:58:57.186 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-02-19T21:58:57.187 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-02-19T21:58:57.248 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-02-19T21:58:57.248 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8" 2024-02-19T21:58:57.248 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-02-19T21:58:57.248 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:58:57.249 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:58:57.249 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-02-19T21:58:57.273 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:58:57.283 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:58:57.288 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:58:57.288 DEBUG:teuthology.orchestra.run.smithi045:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:58:57.291 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:58:57.311 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:58:57.318 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:58:57.318 DEBUG:teuthology.orchestra.run.smithi045:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:58:57.352 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:58:57.392 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:58:57.450 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:58:57.465 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:58:57.465 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:58:57.466 DEBUG:teuthology.orchestra.run.smithi045:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:58:57.466 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:58:57.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:58:57.487 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-02-19T21:58:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:58:57.489 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-02-19T21:58:57.549 INFO:teuthology.orchestra.run.smithi097.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:58:57.550 INFO:teuthology.task.kernel:Running kernel on smithi097: 4.18.0-489.el8.x86_64 2024-02-19T21:58:57.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-02-19T21:58:57.551 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:58:57.551 INFO:teuthology.task.kernel:Running kernel on smithi045: 4.18.0-489.el8.x86_64 2024-02-19T21:58:57.552 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum install -y kernel 2024-02-19T21:59:10.628 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - AppStream 2.5 MB/s | 27 MB 00:10 2024-02-19T21:59:11.455 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - AppStream 2.4 MB/s | 27 MB 00:11 2024-02-19T21:59:17.405 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - BaseOS 9.4 MB/s | 10 MB 00:01 2024-02-19T21:59:18.412 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - BaseOS 3.9 MB/s | 10 MB 00:02 2024-02-19T21:59:19.356 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - Extras 60 kB/s | 18 kB 00:00 2024-02-19T21:59:19.798 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2024-02-19T21:59:20.150 INFO:teuthology.orchestra.run.smithi097.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2024-02-19T21:59:20.278 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras 117 kB/s | 18 kB 00:00 2024-02-19T21:59:20.712 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-19T21:59:20.997 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-02-19T21:59:21.636 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-02-19T21:59:22.214 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2024-02-19T21:59:25.367 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 402 kB/s | 24 kB 00:00 2024-02-19T21:59:26.173 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 437 kB/s | 24 kB 00:00 2024-02-19T21:59:28.322 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:59:28.323 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:59:28.424 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T21:59:28.425 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T21:59:28.425 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T21:59:28.425 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T21:59:28.425 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout:Install 3 Packages 2024-02-19T21:59:28.426 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T21:59:28.427 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 90 M 2024-02-19T21:59:28.427 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 97 M 2024-02-19T21:59:28.427 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T21:59:29.253 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:59:29.253 INFO:teuthology.orchestra.run.smithi045.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:59:29.373 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T21:59:29.374 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T21:59:29.374 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T21:59:29.374 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T21:59:29.374 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout:Install 3 Packages 2024-02-19T21:59:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T21:59:29.376 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 90 M 2024-02-19T21:59:29.376 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 97 M 2024-02-19T21:59:29.376 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T21:59:29.610 INFO:teuthology.orchestra.run.smithi097.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 10 MB/s | 10 MB 00:00 2024-02-19T21:59:30.661 INFO:teuthology.orchestra.run.smithi097.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 18 MB/s | 36 MB 00:02 2024-02-19T21:59:31.189 INFO:teuthology.orchestra.run.smithi097.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-02-19T21:59:31.189 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:59:31.189 INFO:teuthology.orchestra.run.smithi097.stdout:Total 32 MB/s | 90 MB 00:02 2024-02-19T21:59:31.457 INFO:teuthology.orchestra.run.smithi045.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 5.1 MB/s | 10 MB 00:02 2024-02-19T21:59:31.867 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T21:59:31.949 INFO:teuthology.orchestra.run.smithi045.stdout:(2/3): kernel-core-4.18.0-539.el8.x86_64.rpm 17 MB/s | 43 MB 00:02 2024-02-19T21:59:32.284 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T21:59:32.285 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T21:59:33.049 INFO:teuthology.orchestra.run.smithi045.stdout:(3/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 9.9 MB/s | 36 MB 00:03 2024-02-19T21:59:33.050 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:59:33.050 INFO:teuthology.orchestra.run.smithi045.stdout:Total 24 MB/s | 90 MB 00:03 2024-02-19T21:59:33.527 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T21:59:33.551 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T21:59:33.763 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T21:59:34.231 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T21:59:34.231 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T21:59:35.548 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T21:59:35.561 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T21:59:36.337 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T21:59:36.894 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:59:38.490 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T21:59:39.124 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:59:40.460 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:59:40.861 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:59:42.851 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:59:43.224 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:59:44.292 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:59:44.585 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:59:46.904 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:59:47.243 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:06.519 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:06.541 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:06.891 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:06.891 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T22:00:06.891 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T22:00:07.331 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:00:07.332 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:00:07.497 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T22:00:07.710 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T22:00:07.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T22:00:07.711 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T22:00:07.711 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T22:00:07.711 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T22:00:07.711 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T22:00:07.711 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T22:00:07.712 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T22:00:07.921 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T22:00:07.922 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T22:00:07.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.02 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 129.el8 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Group : System Environment/Base 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 9265586 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T22:00:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Relocations : (not relocatable) 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : CentOS BuildSystem 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:00:07.956 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T22:00:07.957 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T22:00:07.958 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 4.18.0-539.el8.x86_64 2024-02-19T22:00:07.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T22:00:09.723 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-02-19T22:00:10.622 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:10.647 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:11.014 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:11.014 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T22:00:11.014 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:00:11.527 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:00:11.559 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-02-19T22:00:11.561 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-02-19T22:00:11.599 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.VHnt4I6mXH 2024-02-19T22:00:11.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VHnt4I6mXH 2024-02-19T22:00:11.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.VHnt4I6mXH 2024-02-19T22:00:11.816 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.VHnt4I6mXH is 6KB 2024-02-19T22:00:11.864 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.VHnt4I6mXH 2024-02-19T22:00:11.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T22:00:11.947 INFO:teuthology.orchestra.run.smithi097.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T22:00:11.947 INFO:teuthology.orchestra.run.smithi097.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T22:00:11.947 INFO:teuthology.orchestra.run.smithi097.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T22:00:11.947 INFO:teuthology.orchestra.run.smithi097.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T22:00:11.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T22:00:12.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-02-19T22:00:12.582 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T22:00:12.816 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T22:00:12.816 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T22:00:12.816 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T22:00:12.816 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T22:00:12.816 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T22:00:12.816 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T22:00:12.816 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi045.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T22:00:12.817 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T22:00:13.037 INFO:teuthology.orchestra.run.smithi045.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T22:00:13.038 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T22:00:13.038 DEBUG:teuthology.orchestra.run.smithi045:> sudo rpm -qi grub2-tools 2024-02-19T22:00:13.071 INFO:teuthology.orchestra.run.smithi045.stdout:Name : grub2-tools 2024-02-19T22:00:13.071 INFO:teuthology.orchestra.run.smithi045.stdout:Epoch : 1 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Version : 2.02 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Release : 129.el8 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Architecture: x86_64 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Group : System Environment/Base 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Size : 9265586 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:License : GPLv3+ 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Relocations : (not relocatable) 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Packager : CentOS BuildSystem 2024-02-19T22:00:13.072 INFO:teuthology.orchestra.run.smithi045.stdout:Vendor : CentOS 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:Summary : Support tools for GRUB. 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:Description : 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:hardware devices. 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:00:13.073 INFO:teuthology.orchestra.run.smithi045.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T22:00:13.074 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T22:00:13.074 INFO:teuthology.task.kernel:Updating grub on smithi045 to boot 4.18.0-539.el8.x86_64 2024-02-19T22:00:13.074 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T22:00:14.137 INFO:teuthology.orchestra.run.smithi045.stderr:Generating grub configuration file ... 2024-02-19T22:00:15.787 INFO:teuthology.orchestra.run.smithi045.stderr:done 2024-02-19T22:00:15.788 DEBUG:teuthology.orchestra.run.smithi045:> mktemp 2024-02-19T22:00:15.816 INFO:teuthology.orchestra.run.smithi045.stdout:/tmp/tmp.kOtbO485qy 2024-02-19T22:00:15.816 DEBUG:teuthology.orchestra.run.smithi045:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kOtbO485qy 2024-02-19T22:00:15.901 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 0666 /tmp/tmp.kOtbO485qy 2024-02-19T22:00:16.073 DEBUG:teuthology.orchestra.remote:smithi045:/tmp/tmp.kOtbO485qy is 6KB 2024-02-19T22:00:16.122 DEBUG:teuthology.orchestra.run.smithi045:> rm -fr /tmp/tmp.kOtbO485qy 2024-02-19T22:00:16.140 DEBUG:teuthology.orchestra.run.smithi045:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T22:00:16.225 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T22:00:16.225 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T22:00:16.225 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T22:00:16.225 INFO:teuthology.orchestra.run.smithi045.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T22:00:16.226 DEBUG:teuthology.orchestra.run.smithi045:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T22:00:16.330 DEBUG:teuthology.orchestra.run.smithi045:> sudo shutdown -r now 2024-02-19T22:00:42.144 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T22:00:42.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:00:42.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:00:46.336 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T22:00:46.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:00:46.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:00.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-02-19T22:01:04.828 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.45 2024-02-19T22:01:09.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:01:09.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:12.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-02-19T22:01:13.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:01:13.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:24.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:01:24.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:42.972 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.97 2024-02-19T22:01:57.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:01:57.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:58.779 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-02-19T22:01:59.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:01:59.804 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T22:01:59.805 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-02-19T22:02:00.292 INFO:teuthology.orchestra.run.smithi097.stdout:4.18.0-539.el8.x86_64 2024-02-19T22:02:00.292 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T22:02:00.292 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T22:02:00.292 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T22:02:01.293 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T22:02:01.293 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T22:02:01.370 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-02-19T22:02:01.382 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:13.832 DEBUG:teuthology.orchestra.remote:timed out 2024-02-19T22:02:25.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:02:25.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:02:26.227 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-02-19T22:02:26.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:02:26.541 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T22:02:26.541 DEBUG:teuthology.orchestra.run.smithi045:> uname -r 2024-02-19T22:02:26.562 INFO:teuthology.orchestra.run.smithi045.stdout:4.18.0-539.el8.x86_64 2024-02-19T22:02:26.562 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T22:02:26.562 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T22:02:26.562 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T22:02:27.563 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T22:02:27.563 DEBUG:teuthology.orchestra.run.smithi045:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T22:02:27.629 INFO:teuthology.orchestra.run.smithi045.stdout:ttyS1 2024-02-19T22:02:27.642 DEBUG:teuthology.parallel:result is None 2024-02-19T22:02:27.642 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T22:02:27.651 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T22:02:27.651 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T22:02:27.654 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T22:02:27.676 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T22:02:27.682 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T22:02:27.690 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T22:02:27.690 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T22:02:27.715 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T22:02:27.760 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T22:02:27.767 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T22:02:27.767 DEBUG:teuthology.orchestra.run.smithi045:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T22:02:27.792 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T22:02:27.827 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T22:02:27.831 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T22:02:27.839 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T22:02:27.842 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T22:02:27.843 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T22:02:27.852 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T22:02:27.852 DEBUG:teuthology.orchestra.run.smithi045:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T22:02:27.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T22:02:27.913 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T22:02:27.923 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T22:02:27.923 DEBUG:teuthology.orchestra.run.smithi045:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T22:02:27.956 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T22:02:27.975 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T22:02:28.057 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T22:02:28.115 DEBUG:teuthology.orchestra.run.smithi045:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T22:02:28.202 DEBUG:teuthology.orchestra.run.smithi045:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T22:02:28.246 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:02:28.246 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T22:02:28.315 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T22:02:28.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T22:02:28.417 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T22:02:28.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T22:02:28.548 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:02:28.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T22:02:28.617 DEBUG:teuthology.orchestra.run.smithi045:> sudo service rsyslog restart 2024-02-19T22:02:28.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-02-19T22:02:28.679 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:02:28.737 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:02:29.150 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T22:02:29.159 INFO:teuthology.task.internal:Starting timer... 2024-02-19T22:02:29.159 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T22:02:29.180 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T22:02:29.188 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T22:02:29.189 DEBUG:teuthology.orchestra.run.smithi045:> sudo service auditd rotate 2024-02-19T22:02:29.276 INFO:teuthology.orchestra.run.smithi045.stdout:Rotating logs: 2024-02-19T22:02:29.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-02-19T22:02:29.366 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-02-19T22:02:29.368 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T22:02:29.368 DEBUG:teuthology.orchestra.run.smithi045:> /usr/sbin/getenforce 2024-02-19T22:02:29.420 INFO:teuthology.orchestra.run.smithi045.stdout:Permissive 2024-02-19T22:02:29.420 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-02-19T22:02:29.459 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-02-19T22:02:29.459 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi045.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi097.front.sepia.ceph.com': 'permissive'} 2024-02-19T22:02:29.460 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:02:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:02:29.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:02:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:02:29.528 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T22:02:29.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo /usr/sbin/setenforce permissive 2024-02-19T22:02:29.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-02-19T22:02:29.596 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T22:02:29.606 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T22:02:29.622 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T22:02:29.623 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi045.front.sepia.ceph.com,smithi097.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T22:08:32.171 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi045.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com')] 2024-02-19T22:08:32.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:08:32.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi045.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:08:32.274 DEBUG:teuthology.orchestra.run.smithi045:> true 2024-02-19T22:08:32.344 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi045.front.sepia.ceph.com' 2024-02-19T22:08:32.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:08:32.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:08:32.447 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-02-19T22:08:32.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-02-19T22:08:32.517 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T22:08:32.527 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T22:08:32.527 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:08:32.528 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:08:32.531 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:08:32.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:08:32.567 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:08:32.583 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:08:32.591 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:08:32.606 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:08:32.650 INFO:teuthology.orchestra.run.smithi045.stderr:sudo: ntpd: command not found 2024-02-19T22:08:32.663 INFO:teuthology.orchestra.run.smithi045.stdout:506 Cannot talk to daemon 2024-02-19T22:08:32.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-02-19T22:08:32.677 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:08:32.683 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-02-19T22:08:32.692 INFO:teuthology.orchestra.run.smithi045.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:08:32.697 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:08:32.711 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:08:32.754 INFO:teuthology.orchestra.run.smithi045.stderr:bash: ntpq: command not found 2024-02-19T22:08:32.757 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:08:32.757 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-02-19T22:08:32.757 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.757 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.757 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.758 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.781 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2024-02-19T22:08:32.784 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:08:32.784 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-02-19T22:08:32.785 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.785 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.785 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.785 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:08:32.786 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:08:32.796 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:08:32.796 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi045.front.sepia.ceph.com 2024-02-19T22:08:32.797 DEBUG:teuthology.orchestra.run.smithi045:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:08:32.797 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi097.front.sepia.ceph.com 2024-02-19T22:08:32.797 DEBUG:teuthology.orchestra.run.smithi097:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:08:33.602 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Feb 2024 10:06:54 PM UTC. 2024-02-19T22:08:33.618 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:30 ago on Mon 19 Feb 2024 10:07:03 PM UTC. 2024-02-19T22:08:35.601 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:08:35.602 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:35.602 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout:Resetting modules: 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:35.603 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:35.798 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout:Resetting modules: 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: container-tools 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:35.799 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:35.846 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:08:36.086 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:08:36.703 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Feb 2024 10:06:54 PM UTC. 2024-02-19T22:08:36.972 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:33 ago on Mon 19 Feb 2024 10:07:03 PM UTC. 2024-02-19T22:08:38.746 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:08:38.781 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout:Installing group/module packages: 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:08:38.782 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:Installing module profiles: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools/common 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:Enabling module streams: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: container-tools rhel8 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2024-02-19T22:08:38.783 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:38.784 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 28 M 2024-02-19T22:08:38.784 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 92 M 2024-02-19T22:08:38.784 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:08:39.143 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout:Installing group/module packages: 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:08:39.182 INFO:teuthology.orchestra.run.smithi045.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:Installing weak dependencies: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:Installing module profiles: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: container-tools/common 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:Enabling module streams: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: container-tools rhel8 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:08:39.183 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:08:39.184 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-02-19T22:08:39.184 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:39.184 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 28 M 2024-02-19T22:08:39.184 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 92 M 2024-02-19T22:08:39.184 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:08:39.450 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 2.7 MB/s | 688 kB 00:00 2024-02-19T22:08:39.500 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 761 kB/s | 38 kB 00:00 2024-02-19T22:08:39.575 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.7 MB/s | 1.0 MB 00:00 2024-02-19T22:08:39.659 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.6 MB/s | 257 kB 00:00 2024-02-19T22:08:39.743 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 2.1 MB/s | 180 kB 00:00 2024-02-19T22:08:39.818 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-02-19T22:08:39.877 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2024-02-19T22:08:40.144 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): aardvark-dns-1.10.0-1.module_el8+871+55 1.9 MB/s | 1.0 MB 00:00 2024-02-19T22:08:40.199 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 700 kB/s | 38 kB 00:00 2024-02-19T22:08:40.224 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): cockpit-podman-83-1.module_el8+847+7863 1.1 MB/s | 688 kB 00:00 2024-02-19T22:08:40.433 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 1.1 MB/s | 257 kB 00:00 2024-02-19T22:08:40.591 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.1 MB/s | 180 kB 00:00 2024-02-19T22:08:40.665 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 338 kB/s | 25 kB 00:00 2024-02-19T22:08:40.742 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2024-02-19T22:08:42.319 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.6 MB/s | 4.3 MB 00:02 2024-02-19T22:08:44.019 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.1 MB/s | 4.3 MB 00:03 2024-02-19T22:08:44.513 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.2 MB/s | 2.5 MB 00:02 2024-02-19T22:08:44.636 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 431 kB/s | 53 kB 00:00 2024-02-19T22:08:44.694 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 704 kB/s | 41 kB 00:00 2024-02-19T22:08:45.028 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.7 MB/s | 8.8 MB 00:05 2024-02-19T22:08:45.269 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): buildah-1.33.5-1.module_el8+885+7da147 1.6 MB/s | 9.7 MB 00:06 2024-02-19T22:08:45.270 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:08:45.270 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.3 MB/s | 28 MB 00:06 2024-02-19T22:08:45.543 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:08:45.563 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:08:45.563 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:08:45.566 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.6 MB/s | 2.5 MB 00:01 2024-02-19T22:08:45.626 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 891 kB/s | 53 kB 00:00 2024-02-19T22:08:45.821 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:08:45.824 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:08:45.845 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 187 kB/s | 41 kB 00:00 2024-02-19T22:08:46.505 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:08:47.417 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): skopeo-1.14.3-0.1.module_el8+885+7da14 1.3 MB/s | 8.8 MB 00:06 2024-02-19T22:08:47.457 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:08:47.526 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): buildah-1.33.5-1.module_el8+885+7da147 1.2 MB/s | 9.7 MB 00:07 2024-02-19T22:08:47.527 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:08:47.527 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.3 MB/s | 28 MB 00:08 2024-02-19T22:08:47.682 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:08:47.808 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:08:47.831 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:08:47.831 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:08:47.844 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:08:48.026 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:08:48.114 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:08:48.117 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:08:48.288 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:08:48.862 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:08:48.865 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:08:49.827 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:08:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:08:50.098 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:08:50.102 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:08:50.288 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:08:50.318 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:08:50.486 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:08:50.755 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:08:50.768 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:08:50.998 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:08:51.240 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:08:51.389 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:08:51.477 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:08:51.903 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:08:51.903 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:08:51.903 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:08:51.903 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:08:51.904 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:08:52.390 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:08:52.390 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:08:52.391 INFO:teuthology.orchestra.run.smithi097.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:08:52.392 INFO:teuthology.orchestra.run.smithi097.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:08:52.392 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:08:52.392 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:08:52.568 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:08:52.668 DEBUG:teuthology.parallel:result is None 2024-02-19T22:08:52.824 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:08:53.056 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:08:53.559 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:08:53.797 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:08:54.032 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:08:54.254 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:08:54.700 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:08:54.700 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:08:54.701 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:08:55.224 INFO:teuthology.orchestra.run.smithi045.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:08:55.225 INFO:teuthology.orchestra.run.smithi045.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:08:55.225 INFO:teuthology.orchestra.run.smithi045.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:08:55.225 INFO:teuthology.orchestra.run.smithi045.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:08:55.225 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:08:55.225 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:08:55.529 DEBUG:teuthology.parallel:result is None 2024-02-19T22:08:55.529 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:08:55.540 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:08:55.541 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi045.front.sepia.ceph.com 2024-02-19T22:08:55.541 DEBUG:teuthology.orchestra.run.smithi045:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:08:55.541 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi097.front.sepia.ceph.com 2024-02-19T22:08:55.542 DEBUG:teuthology.orchestra.run.smithi097:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:08:55.718 DEBUG:teuthology.parallel:result is None 2024-02-19T22:08:55.733 DEBUG:teuthology.parallel:result is None 2024-02-19T22:08:55.733 INFO:teuthology.run_tasks:Running task install... 2024-02-19T22:08:55.744 DEBUG:teuthology.task.install:project ceph 2024-02-19T22:08:55.744 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T22:08:55.744 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:08:55.744 INFO:teuthology.task.install:Using flavor: default 2024-02-19T22:08:55.757 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T22:08:55.757 INFO:teuthology.task.install:extra packages: [] 2024-02-19T22:08:55.758 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:08:55.758 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:08:55.758 INFO:teuthology.packaging:ref: None 2024-02-19T22:08:55.758 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:08:55.759 INFO:teuthology.packaging:branch: None 2024-02-19T22:08:55.759 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:08:55.973 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:08:55.974 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:08:55.976 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:08:55.977 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:08:55.977 INFO:teuthology.packaging:ref: None 2024-02-19T22:08:55.977 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:08:55.977 INFO:teuthology.packaging:branch: None 2024-02-19T22:08:55.977 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:08:55.977 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:08:56.132 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:08:56.132 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:08:56.175 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:08:56.175 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:08:56.175 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:08:56.205 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:08:56.206 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:08:56.206 INFO:teuthology.packaging:ref: None 2024-02-19T22:08:56.206 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:08:56.206 INFO:teuthology.packaging:branch: None 2024-02-19T22:08:56.206 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:08:56.206 DEBUG:teuthology.orchestra.run.smithi045:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:08:56.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:08:56.285 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:08:56.285 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:08:56.335 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:08:56.335 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:08:56.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:08:56.358 DEBUG:teuthology.orchestra.run.smithi045:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:08:56.368 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:08:56.369 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:08:56.369 INFO:teuthology.packaging:ref: None 2024-02-19T22:08:56.369 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:08:56.369 INFO:teuthology.packaging:branch: None 2024-02-19T22:08:56.369 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:08:56.369 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:08:56.426 INFO:teuthology.orchestra.run.smithi045.stdout:check_obsoletes = 1 2024-02-19T22:08:56.427 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum clean all 2024-02-19T22:08:56.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:08:56.520 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:08:56.587 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-02-19T22:08:56.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-02-19T22:08:56.705 INFO:teuthology.orchestra.run.smithi045.stdout:76 files removed 2024-02-19T22:08:56.734 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-radosgw 2024-02-19T22:08:56.857 INFO:teuthology.orchestra.run.smithi097.stdout:76 files removed 2024-02-19T22:08:56.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2024-02-19T22:08:57.237 INFO:teuthology.orchestra.run.smithi045.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-02-19T22:08:57.355 INFO:teuthology.orchestra.run.smithi045.stdout:ceph noarch packages 197 kB/s | 15 kB 00:00 2024-02-19T22:08:57.365 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 980 kB/s | 74 kB 00:00 2024-02-19T22:08:57.472 INFO:teuthology.orchestra.run.smithi045.stdout:ceph source packages 20 kB/s | 1.7 kB 00:00 2024-02-19T22:08:57.483 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2024-02-19T22:08:57.592 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-02-19T22:09:03.533 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - AppStream 4.6 MB/s | 27 MB 00:05 2024-02-19T22:09:09.501 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - BaseOS 8.8 MB/s | 10 MB 00:01 2024-02-19T22:09:10.055 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - AppStream 2.2 MB/s | 27 MB 00:12 2024-02-19T22:09:11.810 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - Extras 25 kB/s | 18 kB 00:00 2024-02-19T22:09:12.469 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2024-02-19T22:09:13.812 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 8 - PowerTools 3.9 MB/s | 4.8 MB 00:01 2024-02-19T22:09:14.872 INFO:teuthology.orchestra.run.smithi097.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-02-19T22:09:15.731 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2024-02-19T22:09:16.347 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - BaseOS 9.4 MB/s | 10 MB 00:01 2024-02-19T22:09:18.314 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2024-02-19T22:09:18.815 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-02-19T22:09:19.548 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 415 kB/s | 24 kB 00:00 2024-02-19T22:09:20.275 INFO:teuthology.orchestra.run.smithi045.stdout:CentOS Stream 8 - PowerTools 3.6 MB/s | 4.8 MB 00:01 2024-02-19T22:09:21.518 INFO:teuthology.orchestra.run.smithi045.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2024-02-19T22:09:23.119 INFO:teuthology.orchestra.run.smithi045.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2024-02-19T22:09:23.128 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:09:23.131 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:09:23.132 INFO:teuthology.orchestra.run.smithi097.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:09:23.133 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout:Install 25 Packages 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2024-02-19T22:09:23.134 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:09:23.135 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 59 M 2024-02-19T22:09:23.135 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:09:23.575 INFO:teuthology.orchestra.run.smithi097.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 35 MB/s | 6.0 MB 00:00 2024-02-19T22:09:23.600 INFO:teuthology.orchestra.run.smithi097.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 998 kB/s | 25 kB 00:00 2024-02-19T22:09:23.667 INFO:teuthology.orchestra.run.smithi097.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 801 kB 00:00 2024-02-19T22:09:23.701 INFO:teuthology.orchestra.run.smithi097.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:09:23.843 INFO:teuthology.orchestra.run.smithi097.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 29 MB/s | 13 MB 00:00 2024-02-19T22:09:23.868 INFO:teuthology.orchestra.run.smithi097.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-19T22:09:23.893 INFO:teuthology.orchestra.run.smithi097.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T22:09:23.919 INFO:teuthology.orchestra.run.smithi097.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-19T22:09:23.944 INFO:teuthology.orchestra.run.smithi097.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 15 MB/s | 387 kB 00:00 2024-02-19T22:09:23.969 INFO:teuthology.orchestra.run.smithi097.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-02-19T22:09:23.995 INFO:teuthology.orchestra.run.smithi097.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T22:09:24.070 INFO:teuthology.orchestra.run.smithi097.stdout:(12/27): librgw2-16.2.4-0.el8.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-02-19T22:09:24.312 INFO:teuthology.orchestra.run.smithi097.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-02-19T22:09:24.337 INFO:teuthology.orchestra.run.smithi097.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 424 kB/s | 145 kB 00:00 2024-02-19T22:09:24.388 INFO:teuthology.orchestra.run.smithi097.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 881 kB/s | 44 kB 00:00 2024-02-19T22:09:24.446 INFO:teuthology.orchestra.run.smithi097.stdout:(16/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 677 kB 00:00 2024-02-19T22:09:24.630 INFO:teuthology.orchestra.run.smithi097.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 818 kB/s | 259 kB 00:00 2024-02-19T22:09:24.655 INFO:teuthology.orchestra.run.smithi097.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 158 kB/s | 42 kB 00:00 2024-02-19T22:09:24.680 INFO:teuthology.orchestra.run.smithi097.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 201 kB/s | 47 kB 00:00 2024-02-19T22:09:24.747 INFO:teuthology.orchestra.run.smithi097.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 684 kB/s | 80 kB 00:00 2024-02-19T22:09:24.873 INFO:teuthology.orchestra.run.smithi097.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 607 kB/s | 116 kB 00:00 2024-02-19T22:09:24.906 INFO:teuthology.orchestra.run.smithi097.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2024-02-19T22:09:24.957 INFO:teuthology.orchestra.run.smithi097.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 701 kB/s | 59 kB 00:00 2024-02-19T22:09:25.007 INFO:teuthology.orchestra.run.smithi097.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 697 kB/s | 181 kB 00:00 2024-02-19T22:09:25.032 INFO:teuthology.orchestra.run.smithi097.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 598 kB/s | 75 kB 00:00 2024-02-19T22:09:25.116 INFO:teuthology.orchestra.run.smithi097.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2024-02-19T22:09:25.174 INFO:teuthology.orchestra.run.smithi097.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2024-02-19T22:09:25.176 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:09:25.177 INFO:teuthology.orchestra.run.smithi097.stdout:Total 29 MB/s | 59 MB 00:02 2024-02-19T22:09:25.247 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:09:25.295 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:09:25.295 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:09:25.655 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:09:25.658 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:09:26.868 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:09:27.090 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:09:27.128 INFO:teuthology.orchestra.run.smithi045.stdout:lab-extras 571 kB/s | 24 kB 00:00 2024-02-19T22:09:27.320 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:09:27.472 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:27.882 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:28.072 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:28.194 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:28.422 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:09:28.623 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:28.680 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:28.915 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:09:29.489 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:09:29.631 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:29.710 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:29.952 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:09:30.122 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:30.193 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:30.386 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:09:30.556 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:09:30.705 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:30.781 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:30.947 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:30.997 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:09:31.000 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrading: 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:09:31.001 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:09:31.002 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout:Install 25 Packages 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout:Upgrade 2 Packages 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:09:31.003 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 59 M 2024-02-19T22:09:31.004 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:09:31.288 INFO:teuthology.orchestra.run.smithi045.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 36 MB/s | 6.0 MB 00:00 2024-02-19T22:09:31.313 INFO:teuthology.orchestra.run.smithi045.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-19T22:09:31.373 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:31.388 INFO:teuthology.orchestra.run.smithi045.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 10 MB/s | 801 kB 00:00 2024-02-19T22:09:31.423 INFO:teuthology.orchestra.run.smithi045.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:09:31.589 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:31.624 INFO:teuthology.orchestra.run.smithi045.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 25 MB/s | 13 MB 00:00 2024-02-19T22:09:31.656 INFO:teuthology.orchestra.run.smithi045.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.4 MB/s | 44 kB 00:00 2024-02-19T22:09:31.682 INFO:teuthology.orchestra.run.smithi045.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T22:09:31.691 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:31.749 INFO:teuthology.orchestra.run.smithi045.stdout:(8/27): librgw2-16.2.4-0.el8.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-02-19T22:09:31.774 INFO:teuthology.orchestra.run.smithi045.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 2.3 MB/s | 213 kB 00:00 2024-02-19T22:09:31.799 INFO:teuthology.orchestra.run.smithi045.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.6 MB/s | 387 kB 00:00 2024-02-19T22:09:31.825 INFO:teuthology.orchestra.run.smithi045.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.1 MB/s | 368 kB 00:00 2024-02-19T22:09:31.850 INFO:teuthology.orchestra.run.smithi045.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-02-19T22:09:31.875 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:09:32.082 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:09:32.140 INFO:teuthology.orchestra.run.smithi045.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2024-02-19T22:09:32.176 INFO:teuthology.orchestra.run.smithi045.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 414 kB/s | 145 kB 00:00 2024-02-19T22:09:32.251 INFO:teuthology.orchestra.run.smithi045.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 587 kB/s | 44 kB 00:00 2024-02-19T22:09:32.272 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:32.350 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:32.468 INFO:teuthology.orchestra.run.smithi045.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 792 kB/s | 259 kB 00:00 2024-02-19T22:09:32.485 INFO:teuthology.orchestra.run.smithi045.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 180 kB/s | 42 kB 00:00 2024-02-19T22:09:32.519 INFO:teuthology.orchestra.run.smithi045.stdout:(18/27): librdkafka-1.6.1-1.el8.x86_64.rpm 1.0 MB/s | 677 kB 00:00 2024-02-19T22:09:32.536 INFO:teuthology.orchestra.run.smithi045.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 699 kB/s | 47 kB 00:00 2024-02-19T22:09:32.538 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:09:32.720 INFO:teuthology.orchestra.run.smithi045.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 343 kB/s | 80 kB 00:00 2024-02-19T22:09:32.795 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:09:32.962 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:09:33.205 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:09:33.303 INFO:teuthology.orchestra.run.smithi045.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 148 kB/s | 116 kB 00:00 2024-02-19T22:09:33.435 INFO:teuthology.orchestra.run.smithi045.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 253 kB/s | 181 kB 00:00 2024-02-19T22:09:33.495 INFO:teuthology.orchestra.run.smithi045.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 306 kB/s | 59 kB 00:00 2024-02-19T22:09:33.579 INFO:teuthology.orchestra.run.smithi045.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2024-02-19T22:09:33.604 INFO:teuthology.orchestra.run.smithi045.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 445 kB/s | 75 kB 00:00 2024-02-19T22:09:33.713 INFO:teuthology.orchestra.run.smithi045.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 30 MB/s | 4.0 MB 00:00 2024-02-19T22:09:33.996 INFO:teuthology.orchestra.run.smithi045.stdout:(27/27): gperftools-libs-2.7-9.el8.x86_64.rpm 210 kB/s | 306 kB 00:01 2024-02-19T22:09:34.001 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:09:34.001 INFO:teuthology.orchestra.run.smithi045.stdout:Total 20 MB/s | 59 MB 00:02 2024-02-19T22:09:34.074 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:09:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:09:34.128 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:09:34.502 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:09:34.506 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:09:35.701 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:09:35.835 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:35.973 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:09:36.047 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:36.213 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:09:36.439 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:36.771 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:36.896 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:37.096 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:37.122 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:37.184 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:37.229 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:37.338 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:37.495 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:09:37.681 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:37.735 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:37.986 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:09:38.580 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:09:38.772 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:38.859 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:39.053 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:09:39.272 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:39.341 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:39.546 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:09:39.732 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:09:39.964 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:40.035 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:40.180 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:40.658 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:40.863 INFO:teuthology.orchestra.run.smithi045.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:40.965 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:41.168 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:09:41.418 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:09:41.605 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:41.684 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:41.884 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:09:42.127 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:09:42.302 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:09:42.505 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:09:45.302 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:45.522 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:46.269 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:46.635 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:46.769 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:46.887 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /run 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-02-19T22:09:56.437 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /dev 2024-02-19T22:09:56.438 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:09:57.794 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:09:57.871 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:09:57.871 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:09:58.088 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:09:58.088 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:58.825 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:09:58.826 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:09:58.827 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.728 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.729 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:09:59.730 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:00.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2024-02-19T22:10:00.784 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:02.339 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:10:02.340 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-02-19T22:10:02.341 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:02.342 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 53 M 2024-02-19T22:10:02.342 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 229 M 2024-02-19T22:10:02.342 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:02.605 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 169 kB 00:00 2024-02-19T22:10:02.655 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.2 MB/s | 203 kB 00:00 2024-02-19T22:10:02.731 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2024-02-19T22:10:02.756 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-02-19T22:10:02.790 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-02-19T22:10:03.407 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 62 MB/s | 52 MB 00:00 2024-02-19T22:10:03.409 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:03.409 INFO:teuthology.orchestra.run.smithi097.stdout:Total 49 MB/s | 53 MB 00:01 2024-02-19T22:10:03.451 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:10:03.472 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:10:03.473 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:10:03.787 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:10:03.790 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:10:04.287 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:10:04.504 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:10:04.726 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:10:04.893 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:10:04.969 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:10:05.175 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:10:05.326 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /sys 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /proc 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /run 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /mnt 2024-02-19T22:10:06.780 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /var/tmp 2024-02-19T22:10:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /home 2024-02-19T22:10:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /tmp 2024-02-19T22:10:06.781 INFO:teuthology.orchestra.run.smithi045.stdout:skipping the directory /dev 2024-02-19T22:10:06.781 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:08.215 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:10:08.295 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:10:08.295 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:10:08.528 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:10:08.529 INFO:teuthology.orchestra.run.smithi045.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:10:09.298 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:10:09.299 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:10:09.300 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:10:10.180 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:10:10.278 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:10:10.278 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:10.278 INFO:teuthology.orchestra.run.smithi045.stdout:Upgraded: 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.279 INFO:teuthology.orchestra.run.smithi045.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:10.280 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:10.440 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:10:10.599 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-test 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:10:10.714 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:11.162 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:11.388 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:44 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:11.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2024-02-19T22:10:12.199 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:13.049 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:10:13.050 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-02-19T22:10:13.051 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:13.052 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 53 M 2024-02-19T22:10:13.052 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 229 M 2024-02-19T22:10:13.052 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:10:13.153 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2024-02-19T22:10:13.187 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.1 MB/s | 203 kB 00:00 2024-02-19T22:10:13.739 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:13.742 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:13.742 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:10:13.742 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:13.742 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:13.742 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:10:13.743 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:10:13.744 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout:Install 32 Packages 2024-02-19T22:10:13.745 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:13.746 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 31 M 2024-02-19T22:10:13.746 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 111 M 2024-02-19T22:10:13.746 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:13.888 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): ceph-test-16.2.4-0.el8.x86_64.rpm 65 MB/s | 52 MB 00:00 2024-02-19T22:10:13.913 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 248 kB/s | 188 kB 00:00 2024-02-19T22:10:13.938 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): socat-1.7.4.1-1.el8.x86_64.rpm 430 kB/s | 323 kB 00:00 2024-02-19T22:10:13.963 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 940 kB/s | 71 kB 00:00 2024-02-19T22:10:13.965 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:13.965 INFO:teuthology.orchestra.run.smithi045.stdout:Total 58 MB/s | 53 MB 00:00 2024-02-19T22:10:14.009 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:10:14.032 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:10:14.032 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:10:14.177 INFO:teuthology.orchestra.run.smithi097.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 120 kB/s | 6.3 kB 00:00 2024-02-19T22:10:14.218 INFO:teuthology.orchestra.run.smithi097.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-02-19T22:10:14.277 INFO:teuthology.orchestra.run.smithi097.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-02-19T22:10:14.302 INFO:teuthology.orchestra.run.smithi097.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-19T22:10:14.359 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:10:14.363 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:10:14.370 INFO:teuthology.orchestra.run.smithi097.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2024-02-19T22:10:14.587 INFO:teuthology.orchestra.run.smithi097.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 49 MB/s | 18 MB 00:00 2024-02-19T22:10:14.604 INFO:teuthology.orchestra.run.smithi097.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 166 kB/s | 39 kB 00:00 2024-02-19T22:10:14.629 INFO:teuthology.orchestra.run.smithi097.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 481 kB/s | 157 kB 00:00 2024-02-19T22:10:14.654 INFO:teuthology.orchestra.run.smithi097.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.1 MB/s | 57 kB 00:00 2024-02-19T22:10:14.713 INFO:teuthology.orchestra.run.smithi097.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.4 MB/s | 457 kB 00:00 2024-02-19T22:10:14.738 INFO:teuthology.orchestra.run.smithi097.stdout:(11/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 686 kB/s | 103 kB 00:00 2024-02-19T22:10:14.911 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:10:14.972 INFO:teuthology.orchestra.run.smithi097.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 784 kB/s | 249 kB 00:00 2024-02-19T22:10:14.997 INFO:teuthology.orchestra.run.smithi097.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 421 kB/s | 109 kB 00:00 2024-02-19T22:10:15.089 INFO:teuthology.orchestra.run.smithi097.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-19T22:10:15.156 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:10:15.190 INFO:teuthology.orchestra.run.smithi097.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 201 kB/s | 44 kB 00:00 2024-02-19T22:10:15.306 INFO:teuthology.orchestra.run.smithi097.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 600 kB/s | 185 kB 00:00 2024-02-19T22:10:15.367 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:10:15.432 INFO:teuthology.orchestra.run.smithi097.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2024-02-19T22:10:15.457 INFO:teuthology.orchestra.run.smithi097.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 470 kB/s | 173 kB 00:00 2024-02-19T22:10:15.533 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:10:15.549 INFO:teuthology.orchestra.run.smithi097.stdout:(19/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.0 MB/s | 384 kB 00:00 2024-02-19T22:10:15.582 INFO:teuthology.orchestra.run.smithi097.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 117 kB/s | 18 kB 00:00 2024-02-19T22:10:15.619 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:10:15.683 INFO:teuthology.orchestra.run.smithi097.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 216 kB/s | 49 kB 00:00 2024-02-19T22:10:15.808 INFO:teuthology.orchestra.run.smithi097.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 227 kB/s | 59 kB 00:00 2024-02-19T22:10:15.825 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:10:15.858 INFO:teuthology.orchestra.run.smithi097.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 89 kB/s | 16 kB 00:00 2024-02-19T22:10:15.925 INFO:teuthology.orchestra.run.smithi097.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 826 kB/s | 283 kB 00:00 2024-02-19T22:10:15.992 INFO:teuthology.orchestra.run.smithi097.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 102 kB/s | 19 kB 00:00 2024-02-19T22:10:16.000 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:10:16.059 INFO:teuthology.orchestra.run.smithi097.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 119 kB/s | 24 kB 00:00 2024-02-19T22:10:16.134 INFO:teuthology.orchestra.run.smithi097.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 140 kB/s | 29 kB 00:00 2024-02-19T22:10:16.193 INFO:teuthology.orchestra.run.smithi097.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 137 kB/s | 27 kB 00:00 2024-02-19T22:10:16.435 INFO:teuthology.orchestra.run.smithi097.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 353 kB/s | 85 kB 00:00 2024-02-19T22:10:16.460 INFO:teuthology.orchestra.run.smithi097.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 637 kB/s | 255 kB 00:00 2024-02-19T22:10:16.485 INFO:teuthology.orchestra.run.smithi097.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 716 kB/s | 251 kB 00:00 2024-02-19T22:10:16.636 INFO:teuthology.orchestra.run.smithi097.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 115 kB/s | 23 kB 00:00 2024-02-19T22:10:16.639 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:16.639 INFO:teuthology.orchestra.run.smithi097.stdout:Total 11 MB/s | 31 MB 00:02 2024-02-19T22:10:16.719 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:10:16.758 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:10:16.758 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:10:17.088 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:10:17.091 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:10:18.247 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:10:18.480 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:10:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:10:18.915 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:10:19.122 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:10:19.328 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:10:19.517 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:10:19.694 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:10:19.929 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:10:20.129 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:10:20.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:10:20.596 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:10:20.852 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:10:21.204 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:10:21.233 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:10:21.448 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:10:21.493 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:10:21.690 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:10:21.751 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:10:21.921 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:10:22.133 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:22.207 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:22.416 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:10:22.429 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph 2024-02-19T22:10:22.621 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:10:22.801 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:10:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:10:23.216 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:23.221 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:10:23.452 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:10:23.682 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:10:24.065 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:10:24.390 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:10:24.753 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:10:24.899 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:10:24.903 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:10:24.904 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:10:24.905 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:Install 32 Packages 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 31 M 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 111 M 2024-02-19T22:10:24.906 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:10:24.924 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:10:25.550 INFO:teuthology.orchestra.run.smithi045.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 82 kB/s | 6.3 kB 00:00 2024-02-19T22:10:25.601 INFO:teuthology.orchestra.run.smithi045.stdout:(2/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-02-19T22:10:25.643 INFO:teuthology.orchestra.run.smithi045.stdout:(3/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 9.3 MB/s | 1.6 MB 00:00 2024-02-19T22:10:25.668 INFO:teuthology.orchestra.run.smithi045.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-19T22:10:25.789 INFO:teuthology.orchestra.run.smithi045.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-19T22:10:25.844 INFO:teuthology.orchestra.run.smithi045.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 895 kB/s | 157 kB 00:00 2024-02-19T22:10:25.886 INFO:teuthology.orchestra.run.smithi045.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 403 kB/s | 39 kB 00:00 2024-02-19T22:10:25.919 INFO:teuthology.orchestra.run.smithi045.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.3 MB/s | 103 kB 00:00 2024-02-19T22:10:26.137 INFO:teuthology.orchestra.run.smithi045.stdout:(9/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 34 MB/s | 18 MB 00:00 2024-02-19T22:10:26.153 INFO:teuthology.orchestra.run.smithi045.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 213 kB/s | 57 kB 00:00 2024-02-19T22:10:26.458 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:10:26.462 INFO:teuthology.orchestra.run.smithi045.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 765 kB/s | 249 kB 00:00 2024-02-19T22:10:26.504 INFO:teuthology.orchestra.run.smithi045.stdout:(12/32): python3-cryptography-3.2.1-7.el8.x86_6 1.8 MB/s | 641 kB 00:00 2024-02-19T22:10:26.521 INFO:teuthology.orchestra.run.smithi045.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-02-19T22:10:26.546 INFO:teuthology.orchestra.run.smithi045.stdout:(14/32): python3-werkzeug-0.12.2-4.el8.noarch.r 730 kB/s | 457 kB 00:00 2024-02-19T22:10:26.588 INFO:teuthology.orchestra.run.smithi045.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 520 kB/s | 44 kB 00:00 2024-02-19T22:10:26.633 INFO:teuthology.orchestra.run.smithi045.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.6 MB/s | 185 kB 00:00 2024-02-19T22:10:26.664 INFO:teuthology.orchestra.run.smithi045.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.4 MB/s | 173 kB 00:00 2024-02-19T22:10:26.673 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:10:26.681 INFO:teuthology.orchestra.run.smithi045.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 234 kB/s | 11 kB 00:00 2024-02-19T22:10:26.706 INFO:teuthology.orchestra.run.smithi045.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 421 kB/s | 18 kB 00:00 2024-02-19T22:10:26.732 INFO:teuthology.orchestra.run.smithi045.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 969 kB/s | 49 kB 00:00 2024-02-19T22:10:26.757 INFO:teuthology.orchestra.run.smithi045.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2024-02-19T22:10:26.782 INFO:teuthology.orchestra.run.smithi045.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.9 MB/s | 384 kB 00:00 2024-02-19T22:10:26.807 INFO:teuthology.orchestra.run.smithi045.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2024-02-19T22:10:26.833 INFO:teuthology.orchestra.run.smithi045.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2024-02-19T22:10:26.858 INFO:teuthology.orchestra.run.smithi045.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 477 kB/s | 24 kB 00:00 2024-02-19T22:10:26.917 INFO:teuthology.orchestra.run.smithi045.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 349 kB/s | 29 kB 00:00 2024-02-19T22:10:26.942 INFO:teuthology.orchestra.run.smithi045.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-02-19T22:10:26.967 INFO:teuthology.orchestra.run.smithi045.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2024-02-19T22:10:26.993 INFO:teuthology.orchestra.run.smithi045.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-19T22:10:27.018 INFO:teuthology.orchestra.run.smithi045.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2024-02-19T22:10:27.043 INFO:teuthology.orchestra.run.smithi045.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2024-02-19T22:10:27.068 INFO:teuthology.orchestra.run.smithi045.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2024-02-19T22:10:27.070 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:27.070 INFO:teuthology.orchestra.run.smithi045.stdout:Total 14 MB/s | 31 MB 00:02 2024-02-19T22:10:27.106 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:10:27.154 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:10:27.196 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:10:27.196 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:10:27.356 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:10:27.528 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:10:27.532 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:10:27.580 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:10:27.790 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:27.808 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:27.939 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:28.328 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:28.328 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:10:28.329 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:10:28.330 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:10:28.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:10:28.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:10:28.331 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:10:28.756 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:10:28.941 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:10:29.132 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:10:29.199 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:10:29.199 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:29.199 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:10:29.199 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:10:29.200 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:10:29.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:10:29.202 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:29.202 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:29.396 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:10:29.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2024-02-19T22:10:29.613 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:10:29.811 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:10:29.984 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:10:30.151 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:10:30.203 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:11 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:30.329 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:10:30.508 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:10:30.728 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:10:30.958 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:10:31.233 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:10:31.638 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:10:31.711 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:31.757 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:31.758 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:10:31.758 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:31.893 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:10:31.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2024-02-19T22:10:32.048 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:10:32.309 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:10:32.500 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:10:32.666 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:32.833 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:10:33.072 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:10:33.277 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:10:33.512 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:10:33.809 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:10:34.054 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:10:34.213 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:34.213 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:34.213 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:34.214 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 71 k 2024-02-19T22:10:34.215 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 290 k 2024-02-19T22:10:34.215 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:34.261 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-02-19T22:10:34.261 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:34.261 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-19T22:10:34.262 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:10:34.268 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:10:34.268 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:10:34.270 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:10:34.273 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:10:34.314 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:10:34.335 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:10:34.722 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:10:34.850 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:35.013 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:10:35.060 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:35.285 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:35.417 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:10:35.653 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:10:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:35.662 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:35.662 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:10:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:10:35.663 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:35.663 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:35.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:10:36.699 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:17 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:37.281 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:10:37.519 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:10:37.980 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:10:38.194 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:10:38.272 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:38.273 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 163 k 2024-02-19T22:10:38.274 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 587 k 2024-02-19T22:10:38.274 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:38.308 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.6 MB/s | 163 kB 00:00 2024-02-19T22:10:38.309 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:38.309 INFO:teuthology.orchestra.run.smithi097.stdout:Total 4.4 MB/s | 163 kB 00:00 2024-02-19T22:10:38.309 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:10:38.320 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:10:38.320 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:10:38.367 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:10:38.370 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:10:38.430 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:10:38.478 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:10:38.600 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:38.618 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:38.672 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:38.739 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:39.086 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:39.164 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:39.164 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:10:39.165 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:10:39.166 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:10:39.167 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:10:39.461 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:39.461 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:39.462 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:10:39.462 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.462 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:39.462 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:39.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2024-02-19T22:10:39.935 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:10:39.935 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:39.935 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:10:39.936 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:10:39.937 INFO:teuthology.orchestra.run.smithi045.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:10:39.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:10:39.938 INFO:teuthology.orchestra.run.smithi045.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:10:39.938 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:39.938 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:40.181 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-base 2024-02-19T22:10:40.442 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:40.978 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:41.944 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:41.991 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:41.992 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:10:41.992 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:42.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:10:42.598 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:42.648 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:42.650 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:10:42.650 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:42.766 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install cephadm 2024-02-19T22:10:42.850 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:43.547 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:44.383 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:44.384 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:44.384 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:10:44.384 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:44.384 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-02-19T22:10:44.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:44.386 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.0 M 2024-02-19T22:10:44.386 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 15 M 2024-02-19T22:10:44.386 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:44.762 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 494 kB/s | 24 kB 00:00 2024-02-19T22:10:44.807 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 95 kB/s | 8.9 kB 00:00 2024-02-19T22:10:44.871 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-02-19T22:10:45.038 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 154 kB/s | 43 kB 00:00 2024-02-19T22:10:45.218 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:45.218 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:10:45.219 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:45.220 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 71 k 2024-02-19T22:10:45.220 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 290 k 2024-02-19T22:10:45.220 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:10:45.260 INFO:teuthology.orchestra.run.smithi045.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-19T22:10:45.261 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:45.261 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-19T22:10:45.261 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:10:45.269 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:10:45.269 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:10:45.271 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:10:45.274 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:10:45.346 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:10:45.780 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 35 kB/s | 34 kB 00:00 2024-02-19T22:10:45.908 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:46.048 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:46.287 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:46.305 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 137 kB/s | 196 kB 00:01 2024-02-19T22:10:46.308 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:46.308 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.5 MB/s | 3.0 MB 00:01 2024-02-19T22:10:46.320 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:10:46.330 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:10:46.330 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:10:46.478 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:10:46.481 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:46.692 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:46.774 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:10:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:10:47.255 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:10:47.473 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:10:47.585 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:10:48.047 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:10:48.535 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:10:48.838 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:48.879 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:10:49.166 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:10:49.569 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:49.570 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:10:49.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:10:50.442 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:10:50.504 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:50.504 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:50.504 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repo Size 2024-02-19T22:10:50.504 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:50.504 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 163 k 2024-02-19T22:10:50.505 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 587 k 2024-02-19T22:10:50.506 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:10:50.550 INFO:teuthology.orchestra.run.smithi045.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.6 MB/s | 163 kB 00:00 2024-02-19T22:10:50.551 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:50.551 INFO:teuthology.orchestra.run.smithi045.stdout:Total 3.5 MB/s | 163 kB 00:00 2024-02-19T22:10:50.551 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:10:50.563 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:10:50.563 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:10:50.613 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:10:50.616 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:10:50.731 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:10:50.962 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:51.403 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:51.775 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:51.993 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:10:51.994 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout:Install 8 Packages 2024-02-19T22:10:51.995 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:10:51.996 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 37 M 2024-02-19T22:10:51.996 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 210 M 2024-02-19T22:10:51.996 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:10:52.016 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr 2024-02-19T22:10:52.569 INFO:teuthology.orchestra.run.smithi097.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 56 MB/s | 8.6 MB 00:00 2024-02-19T22:10:52.810 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:25 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:54.413 INFO:teuthology.orchestra.run.smithi045.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:54.463 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:54.465 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:10:54.465 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:10:54.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:10:55.021 INFO:teuthology.orchestra.run.smithi097.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.5 MB/s | 3.7 MB 00:02 2024-02-19T22:10:55.271 INFO:teuthology.orchestra.run.smithi097.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 901 kB/s | 225 kB 00:00 2024-02-19T22:10:55.382 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:28 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:10:55.738 INFO:teuthology.orchestra.run.smithi097.stdout:(4/8): openblas-0.3.15-4.el8.x86_64.rpm 1.4 MB/s | 4.7 MB 00:03 2024-02-19T22:10:56.347 INFO:teuthology.orchestra.run.smithi097.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.0 MB/s | 645 kB 00:00 2024-02-19T22:10:56.656 INFO:teuthology.orchestra.run.smithi097.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 559 kB/s | 172 kB 00:00 2024-02-19T22:10:56.806 INFO:teuthology.orchestra.run.smithi097.stdout:(7/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.1 MB/s | 4.9 MB 00:04 2024-02-19T22:10:57.055 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:10:57.056 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:10:57.056 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:57.056 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:10:57.056 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-02-19T22:10:57.057 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:10:57.058 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.0 M 2024-02-19T22:10:57.058 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 15 M 2024-02-19T22:10:57.058 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:10:57.754 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 530 kB/s | 24 kB 00:00 2024-02-19T22:10:57.771 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 144 kB/s | 8.9 kB 00:00 2024-02-19T22:10:57.880 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 16 MB/s | 2.7 MB 00:00 2024-02-19T22:10:58.039 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 150 kB/s | 43 kB 00:00 2024-02-19T22:10:58.114 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 98 kB/s | 34 kB 00:00 2024-02-19T22:10:58.372 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 399 kB/s | 196 kB 00:00 2024-02-19T22:10:58.374 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:58.374 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.2 MB/s | 3.0 MB 00:01 2024-02-19T22:10:58.386 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:10:58.397 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:10:58.398 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:10:58.579 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:10:58.582 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:10:58.890 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:10:59.203 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:10:59.483 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:10:59.704 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:10:59.826 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:11:00.890 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:11:01.214 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:11:01.522 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:01.931 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:02.089 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:11:02.877 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:04.546 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:04.547 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:11:04.548 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout:============================================================================================ 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout:Install 8 Packages 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 37 M 2024-02-19T22:11:04.549 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 210 M 2024-02-19T22:11:04.550 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 1.5 MB/s | 14 MB 00:09 2024-02-19T22:11:04.740 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:04.740 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.9 MB/s | 37 MB 00:12 2024-02-19T22:11:04.973 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:04.991 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:04.991 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:05.007 INFO:teuthology.orchestra.run.smithi045.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 52 MB/s | 8.6 MB 00:00 2024-02-19T22:11:05.322 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:05.325 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:05.491 INFO:teuthology.orchestra.run.smithi045.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 7.7 MB/s | 3.7 MB 00:00 2024-02-19T22:11:05.532 INFO:teuthology.orchestra.run.smithi045.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 5.3 MB/s | 225 kB 00:00 2024-02-19T22:11:05.924 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:06.077 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:11:06.237 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:11:06.400 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:11:06.994 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:11:07.160 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:07.224 INFO:teuthology.orchestra.run.smithi045.stdout:(4/8): python3-scipy-1.0.0-21.module_el8+762+77 8.4 MB/s | 14 MB 00:01 2024-02-19T22:11:07.625 INFO:teuthology.orchestra.run.smithi045.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.6 MB/s | 645 kB 00:00 2024-02-19T22:11:07.715 INFO:teuthology.orchestra.run.smithi045.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 172 kB 00:00 2024-02-19T22:11:07.755 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:07.933 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:11:08.621 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:11:08.937 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:11:09.083 INFO:teuthology.orchestra.run.smithi045.stdout:(7/8): openblas-0.3.15-4.el8.x86_64.rpm 1.1 MB/s | 4.7 MB 00:04 2024-02-19T22:11:09.166 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:11:09.317 INFO:teuthology.orchestra.run.smithi045.stdout:(8/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 1.1 MB/s | 4.9 MB 00:04 2024-02-19T22:11:09.318 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:09.318 INFO:teuthology.orchestra.run.smithi045.stdout:Total 7.8 MB/s | 37 MB 00:04 2024-02-19T22:11:09.558 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:09.577 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:09.578 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:10.062 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:10.065 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:10.926 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:11:11.208 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:11.334 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:11:11.506 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:11:11.711 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:11:12.233 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:11:12.343 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:11:12.470 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:11:12.592 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:11:12.787 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:11:13.200 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:13.201 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:13.248 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:13.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:11:13.433 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:11:14.083 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:55 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:14.206 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:11:14.627 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:11:14.900 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:11:15.639 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:11:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:11:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:15.643 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 1.9 M 2024-02-19T22:11:15.644 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 24 M 2024-02-19T22:11:15.644 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:11:15.838 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 399 kB/s | 27 kB 00:00 2024-02-19T22:11:15.897 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 158 kB/s | 20 kB 00:00 2024-02-19T22:11:15.922 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 176 kB/s | 27 kB 00:00 2024-02-19T22:11:16.014 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 940 kB/s | 110 kB 00:00 2024-02-19T22:11:16.040 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 629 kB/s | 126 kB 00:00 2024-02-19T22:11:16.123 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 396 kB/s | 33 kB 00:00 2024-02-19T22:11:16.165 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 389 kB/s | 16 kB 00:00 2024-02-19T22:11:16.182 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 596 kB/s | 155 kB 00:00 2024-02-19T22:11:16.218 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 1.5 MB/s | 81 kB 00:00 2024-02-19T22:11:16.235 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-requests-oauthlib-1.0.0-1.el8. 193 kB/s | 43 kB 00:00 2024-02-19T22:11:16.292 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 871 kB/s | 64 kB 00:00 2024-02-19T22:11:16.344 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 565 kB/s | 61 kB 00:00 2024-02-19T22:11:16.377 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.0 MB/s | 1.2 MB 00:00 2024-02-19T22:11:16.378 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:16.378 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.6 MB/s | 1.9 MB 00:00 2024-02-19T22:11:16.429 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:16.444 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:16.444 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:16.532 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:16.535 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:16.776 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:11:17.066 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:17.310 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:11:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:11:17.735 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:11:17.965 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:11:18.177 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:11:18.197 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:11:18.415 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:11:18.446 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:11:18.735 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:11:18.736 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:11:18.736 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:11:18.773 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:11:19.038 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:11:19.164 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:11:19.164 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:19.164 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:19.164 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:11:19.164 INFO:teuthology.orchestra.run.smithi045.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:19.165 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:19.276 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:11:20.061 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:11:20.075 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:01:53 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:20.340 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:11:20.898 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:11:21.074 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:11:21.224 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:11:21.550 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:11:21.551 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:11:21.735 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:21.737 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:11:21.737 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:21.737 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:21.737 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:21.738 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:21.739 INFO:teuthology.orchestra.run.smithi045.stdout:Install 13 Packages 2024-02-19T22:11:21.739 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:21.739 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 1.9 M 2024-02-19T22:11:21.739 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 24 M 2024-02-19T22:11:21.739 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:21.909 INFO:teuthology.orchestra.run.smithi045.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 585 kB/s | 27 kB 00:00 2024-02-19T22:11:21.967 INFO:teuthology.orchestra.run.smithi045.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 191 kB/s | 20 kB 00:00 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:11:21.990 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:21.991 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:22.026 INFO:teuthology.orchestra.run.smithi045.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 163 kB/s | 27 kB 00:00 2024-02-19T22:11:22.051 INFO:teuthology.orchestra.run.smithi045.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 888 kB/s | 126 kB 00:00 2024-02-19T22:11:22.077 INFO:teuthology.orchestra.run.smithi045.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2024-02-19T22:11:22.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:11:22.294 INFO:teuthology.orchestra.run.smithi045.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 580 kB/s | 155 kB 00:00 2024-02-19T22:11:22.319 INFO:teuthology.orchestra.run.smithi045.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 176 kB/s | 43 kB 00:00 2024-02-19T22:11:22.394 INFO:teuthology.orchestra.run.smithi045.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 217 kB/s | 16 kB 00:00 2024-02-19T22:11:22.436 INFO:teuthology.orchestra.run.smithi045.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 92 kB/s | 33 kB 00:00 2024-02-19T22:11:22.461 INFO:teuthology.orchestra.run.smithi045.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 572 kB/s | 81 kB 00:00 2024-02-19T22:11:22.845 INFO:teuthology.orchestra.run.smithi045.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 160 kB/s | 61 kB 00:00 2024-02-19T22:11:22.864 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:03 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:23.404 INFO:teuthology.orchestra.run.smithi045.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 66 kB/s | 64 kB 00:00 2024-02-19T22:11:24.420 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:11:24.421 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-02-19T22:11:24.422 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:24.423 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 5.9 M 2024-02-19T22:11:24.423 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 24 M 2024-02-19T22:11:24.423 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:11:24.546 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2024-02-19T22:11:24.722 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 121 kB/s | 21 kB 00:00 2024-02-19T22:11:24.811 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.8 MB/s | 578 kB 00:00 2024-02-19T22:11:24.948 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.5 MB/s | 355 kB 00:00 2024-02-19T22:11:25.048 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 206 kB/s | 49 kB 00:00 2024-02-19T22:11:26.431 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.5 MB/s | 4.8 MB 00:01 2024-02-19T22:11:26.432 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:26.432 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.9 MB/s | 5.9 MB 00:02 2024-02-19T22:11:26.486 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:26.496 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:26.497 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:26.606 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:26.609 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:26.893 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:27.190 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:11:27.479 INFO:teuthology.orchestra.run.smithi045.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 235 kB/s | 1.2 MB 00:05 2024-02-19T22:11:27.480 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:27.480 INFO:teuthology.orchestra.run.smithi045.stdout:Total 341 kB/s | 1.9 MB 00:05 2024-02-19T22:11:27.540 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:27.545 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:11:27.557 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:27.557 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:27.652 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:27.656 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:28.189 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:28.411 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:11:28.463 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:11:28.642 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:11:28.793 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:11:28.902 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:11:29.019 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:11:29.147 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:11:29.273 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:29.375 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:11:29.546 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:11:29.615 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:11:29.929 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:11:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:11:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:29.941 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:29.941 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:11:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:11:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:11:29.942 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:29.942 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:30.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2024-02-19T22:11:30.197 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:11:30.820 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:11 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:31.303 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:11:31.755 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:11:31.981 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:11:32.198 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:11:32.364 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:11:32.369 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:32.370 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:32.370 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:32.370 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:32.370 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:32.371 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 968 k 2024-02-19T22:11:32.372 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 3.2 M 2024-02-19T22:11:32.372 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:11:32.607 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:11:32.684 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:11:32.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:11:32.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:11:32.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:11:32.685 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:11:32.774 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 382 kB/s | 83 kB 00:00 2024-02-19T22:11:32.775 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:32.775 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.3 MB/s | 968 kB 00:00 2024-02-19T22:11:32.785 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:32.805 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:32.805 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:32.890 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:32.894 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:33.067 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:33.154 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:11:33.154 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:33.154 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:33.154 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:11:33.154 INFO:teuthology.orchestra.run.smithi045.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:33.155 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:33.280 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:11:33.364 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:11:33.540 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:33.931 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:33.931 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:11:34.067 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:07 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:34.345 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:11:34.346 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:34.346 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:34.346 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:11:34.346 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:34.346 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:34.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2024-02-19T22:11:35.356 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:16 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:35.721 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: Package Arch Version Repository Size 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:11:35.722 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout:Install 6 Packages 2024-02-19T22:11:35.723 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 5.9 M 2024-02-19T22:11:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 24 M 2024-02-19T22:11:35.724 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:36.107 INFO:teuthology.orchestra.run.smithi045.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 575 kB/s | 89 kB 00:00 2024-02-19T22:11:36.341 INFO:teuthology.orchestra.run.smithi045.stdout:(2/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.5 MB/s | 578 kB 00:00 2024-02-19T22:11:36.366 INFO:teuthology.orchestra.run.smithi045.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 82 kB/s | 21 kB 00:00 2024-02-19T22:11:36.691 INFO:teuthology.orchestra.run.smithi045.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 150 kB/s | 49 kB 00:00 2024-02-19T22:11:36.716 INFO:teuthology.orchestra.run.smithi045.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 947 kB/s | 355 kB 00:00 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:11:36.916 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:36.917 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:11:36.917 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:36.917 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 131 k 2024-02-19T22:11:36.917 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 439 k 2024-02-19T22:11:36.917 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:11:36.971 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-02-19T22:11:36.972 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:36.972 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-02-19T22:11:36.972 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:36.978 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:36.978 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:37.027 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:37.031 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:37.142 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:37.405 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:37.692 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:37.808 INFO:teuthology.orchestra.run.smithi045.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.6 MB/s | 4.8 MB 00:01 2024-02-19T22:11:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:37.809 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.8 MB/s | 5.9 MB 00:02 2024-02-19T22:11:37.864 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:37.875 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:37.875 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:37.983 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:37.987 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:38.106 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:38.297 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:38.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2024-02-19T22:11:38.551 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:11:38.874 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:11:39.118 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:20 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:39.887 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:11:40.230 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:11:40.478 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:11:40.678 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:40.712 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:40.725 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:40.726 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:40.726 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:40.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:11:41.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:41.427 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:41.589 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:22 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:41.598 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install ceph-fuse 2024-02-19T22:11:42.395 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:15 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:43.157 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:11:43.158 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:43.159 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 24 k 2024-02-19T22:11:43.159 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 94 k 2024-02-19T22:11:43.159 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:11:43.279 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 202 kB/s | 24 kB 00:00 2024-02-19T22:11:43.279 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:43.280 INFO:teuthology.orchestra.run.smithi097.stdout:Total 200 kB/s | 24 kB 00:00 2024-02-19T22:11:43.280 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:11:43.283 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:11:43.283 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:11:43.285 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:11:43.288 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:11:43.341 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:11:43.571 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:43.840 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:44.052 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout:Installing dependencies: 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:11:44.053 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:Install 2 Packages 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 968 k 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 3.2 M 2024-02-19T22:11:44.054 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:44.186 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:44.186 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:44.186 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:11:44.186 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:44.187 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:11:44.187 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:44.301 INFO:teuthology.orchestra.run.smithi045.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2024-02-19T22:11:44.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2024-02-19T22:11:44.468 INFO:teuthology.orchestra.run.smithi045.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 377 kB/s | 83 kB 00:00 2024-02-19T22:11:44.469 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:44.469 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.3 MB/s | 968 kB 00:00 2024-02-19T22:11:44.480 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:44.502 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:44.503 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:44.600 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:44.603 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:44.789 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:45.034 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:11:45.059 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:45.196 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:45.625 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:45.625 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:46.016 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:46.277 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados-devel 2024-02-19T22:11:46.571 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:46.619 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:46.620 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:46.621 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:46.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2024-02-19T22:11:47.074 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:20 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:47.486 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:48.741 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:48.742 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:48.743 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:11:48.743 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:48.743 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 131 k 2024-02-19T22:11:48.743 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 439 k 2024-02-19T22:11:48.743 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:48.787 INFO:teuthology.orchestra.run.smithi045.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 131 kB 00:00 2024-02-19T22:11:48.787 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:48.788 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.9 MB/s | 131 kB 00:00 2024-02-19T22:11:48.788 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:48.794 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:48.794 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:48.846 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:48.850 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:48.967 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:48.992 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:49.039 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:49.041 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:49.041 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:49.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2024-02-19T22:11:49.186 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:49.497 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:49.885 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:49.904 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:30 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:50.090 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs2 2024-02-19T22:11:50.887 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:51.417 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:51.464 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:51.465 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:51.465 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:51.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2024-02-19T22:11:52.331 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:33 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:52.498 INFO:teuthology.orchestra.run.smithi045.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:52.550 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:11:52.551 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:52.646 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install libcephfs-devel 2024-02-19T22:11:53.434 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:53.858 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:53.905 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:53.906 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:53.906 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:54.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2024-02-19T22:11:54.766 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:55.098 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:55.098 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:55.098 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:55.098 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:55.098 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:55.099 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 24 k 2024-02-19T22:11:55.100 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 94 k 2024-02-19T22:11:55.100 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:11:55.141 INFO:teuthology.orchestra.run.smithi045.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 585 kB/s | 24 kB 00:00 2024-02-19T22:11:55.142 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:55.142 INFO:teuthology.orchestra.run.smithi045.stdout:Total 567 kB/s | 24 kB 00:00 2024-02-19T22:11:55.142 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:11:55.146 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:11:55.146 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:11:55.149 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:11:55.152 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:11:55.204 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:11:55.450 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:55.732 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:11:56.082 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:56.198 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librados2 2024-02-19T22:11:56.258 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:56.304 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:56.306 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:56.306 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:56.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2024-02-19T22:11:56.984 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:29 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:57.164 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:38 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:11:58.606 INFO:teuthology.orchestra.run.smithi045.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:58.656 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:11:58.658 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:11:58.658 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:11:58.669 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:58.715 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:11:58.716 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-02-19T22:11:58.716 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:11:58.754 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install librbd1 2024-02-19T22:11:58.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2024-02-19T22:11:59.542 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:32 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:11:59.573 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:12:01.142 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:12:01.142 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 89 k 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 298 k 2024-02-19T22:12:01.143 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:12:01.148 INFO:teuthology.orchestra.run.smithi045.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:12:01.186 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-02-19T22:12:01.186 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:01.186 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-19T22:12:01.187 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:12:01.196 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:12:01.196 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:12:01.199 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:01.200 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:12:01.201 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:01.245 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:12:01.248 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:12:01.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rados 2024-02-19T22:12:01.350 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:12:01.535 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:01.822 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:02.090 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:02.183 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:12:02.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2024-02-19T22:12:03.195 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:44 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:12:03.725 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:12:03.775 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:12:03.777 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:03.873 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rgw 2024-02-19T22:12:04.679 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:37 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:04.738 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:12:04.739 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:04.740 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.7 M 2024-02-19T22:12:04.740 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 18 M 2024-02-19T22:12:04.740 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:12:04.823 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2024-02-19T22:12:04.824 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:04.824 INFO:teuthology.orchestra.run.smithi097.stdout:Total 44 MB/s | 3.7 MB 00:00 2024-02-19T22:12:04.824 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:12:04.835 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:12:04.836 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:12:04.898 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:12:04.901 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:12:05.393 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:12:05.569 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:05.989 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:06.293 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:12:06.344 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:06.346 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:12:06.346 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:06.426 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:06.426 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:12:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:06.427 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:06.427 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:12:06.445 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-cephfs 2024-02-19T22:12:06.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2024-02-19T22:12:07.240 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:40 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:07.429 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:02:48 ago on Mon 19 Feb 2024 10:09:19 PM UTC. 2024-02-19T22:12:08.866 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:12:08.917 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:08.918 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:12:08.918 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:08.973 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-02-19T22:12:08.974 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:08.975 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 173 k 2024-02-19T22:12:08.975 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 649 k 2024-02-19T22:12:08.975 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-02-19T22:12:09.021 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install python3-rbd 2024-02-19T22:12:09.026 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.3 MB/s | 173 kB 00:00 2024-02-19T22:12:09.027 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:09.027 INFO:teuthology.orchestra.run.smithi097.stdout:Total 3.2 MB/s | 173 kB 00:00 2024-02-19T22:12:09.027 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-02-19T22:12:09.038 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-02-19T22:12:09.038 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-02-19T22:12:09.087 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-02-19T22:12:09.090 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-02-19T22:12:09.201 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-02-19T22:12:09.435 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:09.723 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:09.822 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:42 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:12:10.106 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-02-19T22:12:10.309 DEBUG:teuthology.parallel:result is None 2024-02-19T22:12:11.435 INFO:teuthology.orchestra.run.smithi045.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:12:11.486 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:11.487 INFO:teuthology.orchestra.run.smithi045.stdout:Nothing to do. 2024-02-19T22:12:11.487 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:11.586 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-fuse 2024-02-19T22:12:12.372 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:45 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:14.048 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:14.049 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:12:14.050 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:14.050 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 89 k 2024-02-19T22:12:14.050 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 298 k 2024-02-19T22:12:14.050 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:12:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.6 MB/s | 89 kB 00:00 2024-02-19T22:12:14.106 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:Total 1.5 MB/s | 89 kB 00:00 2024-02-19T22:12:14.107 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:12:14.117 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:12:14.118 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:12:14.170 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:12:14.173 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:12:14.282 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:12:14.474 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:14.804 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:15.237 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:15.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:15.238 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:12:15.238 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:15.238 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:15.238 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:15.583 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-mirror 2024-02-19T22:12:16.372 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:49 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:18.036 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:12:18.037 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:18.038 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 3.7 M 2024-02-19T22:12:18.038 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 18 M 2024-02-19T22:12:18.038 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:12:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2024-02-19T22:12:18.128 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:Total 40 MB/s | 3.7 MB 00:00 2024-02-19T22:12:18.129 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:12:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:12:18.142 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:12:18.211 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:12:18.214 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:12:18.707 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:12:18.907 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:19.378 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:19.779 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:12:19.779 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:19.780 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:19.780 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:20.014 DEBUG:teuthology.orchestra.run.smithi045:> sudo yum -y install rbd-nbd 2024-02-19T22:12:20.818 INFO:teuthology.orchestra.run.smithi045.stdout:Last metadata expiration check: 0:02:53 ago on Mon 19 Feb 2024 10:09:27 PM UTC. 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout:Dependencies resolved. 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout: Package Architecture Version Repository Size 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout:Installing: 2024-02-19T22:12:22.496 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction Summary 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:================================================================================ 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:Install 1 Package 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:Total download size: 173 k 2024-02-19T22:12:22.497 INFO:teuthology.orchestra.run.smithi045.stdout:Installed size: 649 k 2024-02-19T22:12:22.498 INFO:teuthology.orchestra.run.smithi045.stdout:Downloading Packages: 2024-02-19T22:12:22.557 INFO:teuthology.orchestra.run.smithi045.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 173 kB 00:00 2024-02-19T22:12:22.557 INFO:teuthology.orchestra.run.smithi045.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:12:22.558 INFO:teuthology.orchestra.run.smithi045.stdout:Total 2.8 MB/s | 173 kB 00:00 2024-02-19T22:12:22.558 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction check 2024-02-19T22:12:22.569 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction check succeeded. 2024-02-19T22:12:22.570 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction test 2024-02-19T22:12:22.625 INFO:teuthology.orchestra.run.smithi045.stdout:Transaction test succeeded. 2024-02-19T22:12:22.628 INFO:teuthology.orchestra.run.smithi045.stdout:Running transaction 2024-02-19T22:12:22.751 INFO:teuthology.orchestra.run.smithi045.stdout: Preparing : 1/1 2024-02-19T22:12:22.966 INFO:teuthology.orchestra.run.smithi045.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:23.248 INFO:teuthology.orchestra.run.smithi045.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout:Installed: 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout: 2024-02-19T22:12:23.645 INFO:teuthology.orchestra.run.smithi045.stdout:Complete! 2024-02-19T22:12:23.870 DEBUG:teuthology.parallel:result is None 2024-02-19T22:12:23.870 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:12:23.870 INFO:teuthology.packaging:ref: None 2024-02-19T22:12:23.870 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:12:23.870 INFO:teuthology.packaging:branch: None 2024-02-19T22:12:23.870 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:12:23.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:12:24.027 DEBUG:teuthology.orchestra.run.smithi045:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:12:24.050 INFO:teuthology.orchestra.run.smithi045.stdout:16.2.4-0.el8 2024-02-19T22:12:24.051 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:12:24.051 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:12:24.053 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:12:24.053 INFO:teuthology.packaging:ref: None 2024-02-19T22:12:24.053 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:12:24.053 INFO:teuthology.packaging:branch: None 2024-02-19T22:12:24.053 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:12:24.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:12:24.284 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:12:24.310 INFO:teuthology.orchestra.run.smithi097.stdout:16.2.4-0.el8 2024-02-19T22:12:24.311 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:12:24.311 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:12:24.312 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T22:12:24.313 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:12:24.313 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:12:24.342 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:12:24.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:12:24.383 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T22:12:24.384 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:12:24.384 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:12:24.412 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:12:24.479 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:12:24.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:12:24.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:12:24.579 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T22:12:24.579 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:12:24.579 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:12:24.607 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:12:24.674 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:12:24.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:12:24.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:12:24.775 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T22:12:24.776 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:12:24.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:12:24.802 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:12:24.870 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:12:24.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:12:24.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:12:24.968 INFO:teuthology.run_tasks:Running task print... 2024-02-19T22:12:24.980 INFO:teuthology.task.print:**** done install task... 2024-02-19T22:12:24.980 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T22:12:25.057 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:12:25.057 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T22:12:25.058 INFO:tasks.cephadm:Cluster fsid is f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:25.058 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T22:12:25.058 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T22:12:25.058 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi045': '172.21.15.45', 'mon.smithi097': '172.21.15.97'} 2024-02-19T22:12:25.058 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T22:12:25.059 DEBUG:teuthology.orchestra.run.smithi045:> sudo hostname $(hostname -s) 2024-02-19T22:12:25.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-02-19T22:12:25.118 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T22:12:25.118 DEBUG:teuthology.orchestra.run.smithi045:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.264 INFO:teuthology.orchestra.run.smithi045.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:12 /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.264 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.471 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:12 /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.472 DEBUG:teuthology.orchestra.run.smithi045:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.493 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:12:25.524 DEBUG:teuthology.orchestra.run.smithi045:> sudo mkdir -p /etc/ceph 2024-02-19T22:12:25.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-02-19T22:12:25.592 DEBUG:teuthology.orchestra.run.smithi045:> sudo chmod 777 /etc/ceph 2024-02-19T22:12:25.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-02-19T22:12:25.661 INFO:tasks.cephadm:Writing seed config... 2024-02-19T22:12:25.662 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T22:12:25.662 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T22:12:25.662 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T22:12:25.662 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T22:12:25.662 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T22:12:25.663 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T22:12:25.664 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T22:12:25.665 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T22:12:25.665 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T22:12:25.665 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T22:12:25.665 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T22:12:25.665 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T22:12:25.665 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:12:25.666 DEBUG:teuthology.orchestra.run.smithi045:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T22:12:25.691 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f6020aec-cf73-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T22:12:25.692 DEBUG:teuthology.orchestra.run.smithi045:mon.smithi045> sudo journalctl -f -n 0 -u ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service 2024-02-19T22:12:25.734 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T22:12:25.735 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid f6020aec-cf73-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.45 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:12:25.762 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:-- Logs begin at Mon 2024-02-19 22:01:36 UTC. -- 2024-02-19T22:12:26.006 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:12:27.052 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:12:27.063 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying podman|docker is present... 2024-02-19T22:12:27.063 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:12:27.088 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying lvm2 is present... 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying time synchronization is in place... 2024-02-19T22:12:27.100 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:12:27.108 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:12:27.109 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:12:27.115 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:12:27.115 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:12:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: enabled 2024-02-19T22:12:27.122 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:12:27.128 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: active 2024-02-19T22:12:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:12:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:Repeating the final host check... 2024-02-19T22:12:27.129 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:12:27.151 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:12:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:podman|docker (/bin/podman) is present 2024-02-19T22:12:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl is present 2024-02-19T22:12:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:lvcreate is present 2024-02-19T22:12:27.162 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:12:27.168 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:12:27.169 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:12:27.174 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:12:27.175 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:12:27.181 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: enabled 2024-02-19T22:12:27.181 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:12:27.187 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: active 2024-02-19T22:12:27.187 INFO:teuthology.orchestra.run.smithi045.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:12:27.187 INFO:teuthology.orchestra.run.smithi045.stderr:Host looks OK 2024-02-19T22:12:27.188 INFO:teuthology.orchestra.run.smithi045.stderr:Cluster fsid: f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:27.188 INFO:teuthology.orchestra.run.smithi045.stderr:Acquiring lock 139829399501288 on /run/cephadm/f6020aec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:12:27.188 INFO:teuthology.orchestra.run.smithi045.stderr:Lock 139829399501288 acquired on /run/cephadm/f6020aec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:12:27.188 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying IP 172.21.15.45 port 3300 ... 2024-02-19T22:12:27.189 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying IP 172.21.15.45 port 6789 ... 2024-02-19T22:12:27.189 INFO:teuthology.orchestra.run.smithi045.stderr:Base mon IP is 172.21.15.45, final addrv is [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-02-19T22:12:27.189 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip route ls 2024-02-19T22:12:27.191 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.45 metric 100 2024-02-19T22:12:27.192 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.45 metric 100 2024-02-19T22:12:27.192 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T22:12:27.195 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T22:12:27.195 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-02-19T22:12:27.195 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1766sec hoplimit 64 pref medium 2024-02-19T22:12:27.195 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:12be/64 scope link 2024-02-19T22:12:27.198 INFO:teuthology.orchestra.run.smithi045.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:12:27.199 INFO:teuthology.orchestra.run.smithi045.stderr:Mon IP 172.21.15.45 is in CIDR network 172.21.0.0/20 2024-02-19T22:12:27.199 INFO:teuthology.orchestra.run.smithi045.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T22:12:27.201 INFO:teuthology.orchestra.run.smithi045.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:12:27.201 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T22:12:27.297 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:12:33.901 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Getting image source signatures 2024-02-19T22:12:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T22:12:33.908 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T22:12:46.319 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:12:46.444 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T22:12:46.790 INFO:teuthology.orchestra.run.smithi045.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:12:46.802 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T22:12:47.614 INFO:teuthology.orchestra.run.smithi045.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:12:48.661 INFO:teuthology.orchestra.run.smithi045.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:12:48.661 INFO:teuthology.orchestra.run.smithi045.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T22:12:48.662 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T22:12:49.324 INFO:teuthology.orchestra.run.smithi045.stderr:stat: 167 167 2024-02-19T22:12:50.028 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial keys... 2024-02-19T22:12:50.029 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:50.579 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQBi0tNlkPPwHRAAFNyBx/dFdEJlwFJhyLqNIA== 2024-02-19T22:12:51.258 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:51.877 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQBj0tNltCMQLhAAIU5OqKDfaYigV1lvfSFfig== 2024-02-19T22:12:52.549 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:53.076 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-authtool: AQBk0tNlAihZORAALzjz5e6hSp8MdPTRImGUjA== 2024-02-19T22:12:53.686 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial monmap... 2024-02-19T22:12:53.686 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpmnh2c12f:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid f6020aec-cf73-11ee-95bb-87774f69a715 --addv smithi045 [v2:172.21.15.45:3300,v1:172.21.15.45:6789] /tmp/monmap 2024-02-19T22:12:54.198 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:12:54.198 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:54.198 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:12:54.745 INFO:teuthology.orchestra.run.smithi045.stderr:monmaptool for smithi045 [v2:172.21.15.45:3300,v1:172.21.15.45:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:12:54.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: set fsid to f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:54.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:12:54.745 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:12:54.745 INFO:teuthology.orchestra.run.smithi045.stderr:Creating mon... 2024-02-19T22:12:54.746 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/mon.smithi045:/var/lib/ceph/mon/ceph-smithi045:z -v /tmp/ceph-tmpgzn1hzi7:/tmp/keyring:z -v /tmp/ceph-tmpmnh2c12f:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi045 --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T22:12:55.348 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.245+0000 7f72b398f700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 1 imported monmap: 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: fsid f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T22:12:54.148702+0000 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: created 2024-02-19T22:12:54.148702+0000 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon.smithi045 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 0 /usr/bin/ceph-mon: set fsid to f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:12:55.349 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: DB SUMMARY 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi045/store.db dir, Total Num: 0, files: 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi045/store.db: 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.env: 0x55d0d3d721c0 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.info_log: 0x55d0d56f6ec0 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:12:55.350 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi045/store.db 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:12:55.351 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.write_buffer_manager: 0x55d0d56fd530 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:12:55.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.row_cache: None 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:12:55.353 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:12:55.354 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:12:55.355 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.246+0000 7f72b398f700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000001 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.merge_operator: 2024-02-19T22:12:55.356 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d0d560fcb8) 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T22:12:55.357 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache: 0x55d0d5644d10 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:12:55.358 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:12:55.359 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:12:55.360 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:12:55.361 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:12:55.362 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.247+0000 7f72b398f700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.248+0000 7f72b398f700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.363 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.248+0000 7f72b398f700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.248+0000 7f72b398f700 4 rocksdb: DB pointer 0x55d0d5709800 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.249+0000 7f729c828700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.249+0000 7f729c828700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:55.364 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:55.365 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.366 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:55.367 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.250+0000 7f72b398f700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T22:12:55.368 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.250+0000 7f72b398f700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T22:12:55.368 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:55.250+0000 7f72b398f700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi045 for mon.smithi045 2024-02-19T22:12:55.933 INFO:teuthology.orchestra.run.smithi045.stderr:create mon.smithi045 on 2024-02-19T22:12:55.934 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:55.940 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl disable ceph.target 2024-02-19T22:12:55.948 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T22:12:56.040 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph.target 2024-02-19T22:12:56.047 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T22:12:56.131 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph.target 2024-02-19T22:12:56.138 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:12:56.144 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:12:56.144 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:12:56.227 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:12:56.235 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl daemon-reload 2024-02-19T22:12:56.324 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:12:56.351 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl reset-failed ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:12:56.357 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to reset failed state of unit ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service: Unit ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service not loaded. 2024-02-19T22:12:56.357 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:12:56.364 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target.wants/ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service → /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:12:56.455 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:12:56.497 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:56 smithi045 systemd[1]: Starting Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:12:57.032 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:12:57.040 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2024-02-19T22:12:57.040 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:12:57.048 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:12:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2024-02-19T22:12:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:12:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:12:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mon to start... 2024-02-19T22:12:57.049 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mon... 2024-02-19T22:12:57.050 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/mon.smithi045:/var/lib/ceph/mon/ceph-smithi045:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T22:12:57.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:56 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.983856+0000 mon.smithi045 (mon.0) 0 : [INF] mkfs f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:57.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:56 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.968617+0000 mon.smithi045 (mon.0) 1 : cluster [INF] mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-02-19T22:12:57.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 bash[37666]: 51d9bdb1d7ce73e36ed49ba0a278077147dfc75fa71795b74d2c0c6033175a91 2024-02-19T22:12:57.306 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 systemd[1]: Started Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:12:57.614 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.612+0000 7fe59931c700 1 Processor -- start 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.612+0000 7fe59931c700 1 -- start start 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.612+0000 7fe59931c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.612+0000 7fe59931c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5940fb6a0 con 0x7fe5940fad90 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55506/0 (socket says 172.21.15.45:55506) 2024-02-19T22:12:57.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe592ffd700 1 -- 172.21.15.45:0/1740738479 learned_addr learned my addr 172.21.15.45:0/1740738479 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe592ffd700 1 -- 172.21.15.45:0/1740738479 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940fb7e0 con 0x7fe5940fad90 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe592ffd700 1 --2- 172.21.15.45:0/1740738479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe58400b5f0 tx=0x7fe584006ee0).ready entity=mon.0 client_cookie=f07364728dc56960 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe591ffb700 1 -- 172.21.15.45:0/1740738479 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58400d600 con 0x7fe5940fad90 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe591ffb700 1 -- 172.21.15.45:0/1740738479 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe58400d760 con 0x7fe5940fad90 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.613+0000 7fe591ffb700 1 -- 172.21.15.45:0/1740738479 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58400da30 con 0x7fe5940fad90 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 -- 172.21.15.45:0/1740738479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 msgr2=0x7fe5940fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 --2- 172.21.15.45:0/1740738479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe58400b5f0 tx=0x7fe584006ee0).stop 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 -- 172.21.15.45:0/1740738479 shutdown_connections 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 --2- 172.21.15.45:0/1740738479 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe5940fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 -- 172.21.15.45:0/1740738479 >> 172.21.15.45:0/1740738479 conn(0x7fe5940f6530 msgr2=0x7fe5940f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 -- 172.21.15.45:0/1740738479 shutdown_connections 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.614+0000 7fe59931c700 1 -- 172.21.15.45:0/1740738479 wait complete. 2024-02-19T22:12:57.617 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.615+0000 7fe59931c700 1 Processor -- start 2024-02-19T22:12:57.619 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.615+0000 7fe59931c700 1 -- start start 2024-02-19T22:12:57.619 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.615+0000 7fe59931c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.615+0000 7fe59931c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe594070470 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.615+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe592ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55518/0 (socket says 172.21.15.45:55518) 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe592ffd700 1 -- 172.21.15.45:0/1034312272 learned_addr learned my addr 172.21.15.45:0/1034312272 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe592ffd700 1 -- 172.21.15.45:0/1034312272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58400b200 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe592ffd700 1 --2- 172.21.15.45:0/1034312272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe584006c70 tx=0x7fe584016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe584011b40 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe584011ca0 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe594070670 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe59410d720 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.616+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe58401d430 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.617+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe58400f070 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.617+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe584027460 con 0x7fe5940fad90 2024-02-19T22:12:57.620 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.618+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574004fa0 con 0x7fe5940fad90 2024-02-19T22:12:57.621 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.620+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe584016070 con 0x7fe5940fad90 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.689+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe574005940 con 0x7fe5940fad90 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.689+0000 7fe57b7fe700 1 -- 172.21.15.45:0/1034312272 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe584035160 con 0x7fe5940fad90 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: cluster: 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: id: f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.693 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: services: 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi045 (age 0.707268s) 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: data: 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: pgs: 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 msgr2=0x7fe59410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 --2- 172.21.15.45:0/1034312272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe584006c70 tx=0x7fe584016000).stop 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 shutdown_connections 2024-02-19T22:12:57.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 --2- 172.21.15.45:0/1034312272 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5940fad90 0x7fe59410d070 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 >> 172.21.15.45:0/1034312272 conn(0x7fe5940f6530 msgr2=0x7fe5940fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 shutdown_connections 2024-02-19T22:12:57.695 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.690+0000 7fe59931c700 1 -- 172.21.15.45:0/1034312272 wait complete. 2024-02-19T22:12:58.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.983415+0000 mon.smithi045 (mon.0) 2 : cluster [INF] mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-02-19T22:12:58.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.983544+0000 mon.smithi045 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} 2024-02-19T22:12:58.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.984948+0000 mon.smithi045 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T22:12:58.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.986770+0000 mon.smithi045 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:12:58.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: cluster 2024-02-19T22:12:56.987129+0000 mon.smithi045 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:12:58.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: audit 2024-02-19T22:12:57.690654+0000 mon.smithi045 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.45:0/1034312272' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T22:12:58.413 INFO:teuthology.orchestra.run.smithi045.stderr:mon is available 2024-02-19T22:12:58.413 INFO:teuthology.orchestra.run.smithi045.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T22:12:58.413 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z -v /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/mon.smithi045:/var/lib/ceph/mon/ceph-smithi045:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi045/config 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.941+0000 7f1aa51d5700 1 Processor -- start 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.941+0000 7f1aa51d5700 1 -- start start 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.941+0000 7f1aa51d5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.941+0000 7f1aa51d5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa00fc8d0 con 0x7f1aa00703d0 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55520/0 (socket says 172.21.15.45:55520) 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9ed9d700 1 -- 172.21.15.45:0/2088289274 learned_addr learned my addr 172.21.15.45:0/2088289274 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9ed9d700 1 -- 172.21.15.45:0/2088289274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa01013d0 con 0x7f1aa00703d0 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9ed9d700 1 --2- 172.21.15.45:0/2088289274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1a9000b800 tx=0x7f1a90006ee0).ready entity=mon.0 client_cookie=a43acaa40eb3350e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:58.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9dd9b700 1 -- 172.21.15.45:0/2088289274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a90009740 con 0x7f1aa00703d0 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9dd9b700 1 -- 172.21.15.45:0/2088289274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a900098a0 con 0x7f1aa00703d0 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.942+0000 7f1a9dd9b700 1 -- 172.21.15.45:0/2088289274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a90009b70 con 0x7f1aa00703d0 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 -- 172.21.15.45:0/2088289274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 msgr2=0x7f1aa0100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 --2- 172.21.15.45:0/2088289274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1a9000b800 tx=0x7f1a90006ee0).stop 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 -- 172.21.15.45:0/2088289274 shutdown_connections 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 --2- 172.21.15.45:0/2088289274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa0100ee0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 -- 172.21.15.45:0/2088289274 >> 172.21.15.45:0/2088289274 conn(0x7f1aa00f6520 msgr2=0x7f1aa00f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 -- 172.21.15.45:0/2088289274 shutdown_connections 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.943+0000 7f1aa51d5700 1 -- 172.21.15.45:0/2088289274 wait complete. 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1aa51d5700 1 Processor -- start 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1aa51d5700 1 -- start start 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1aa51d5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:58.947 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1aa51d5700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa010ca70 con 0x7f1aa00703d0 2024-02-19T22:12:58.948 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1a9ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:58.948 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1a9ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55528/0 (socket says 172.21.15.45:55528) 2024-02-19T22:12:58.948 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.944+0000 7f1a9ed9d700 1 -- 172.21.15.45:0/919047349 learned_addr learned my addr 172.21.15.45:0/919047349 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:12:58.948 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a9ed9d700 1 -- 172.21.15.45:0/919047349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a9000b200 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a9ed9d700 1 --2- 172.21.15.45:0/919047349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1a90006c70 tx=0x7f1a90016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9001d430 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9001d590 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa010cc70 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9001d8a0 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa010d100 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1a9000f070 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.945+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1a900277d0 con 0x7f1aa00703d0 2024-02-19T22:12:58.949 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.946+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa0045ad0 con 0x7f1aa00703d0 2024-02-19T22:12:58.950 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.949+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1a90016070 con 0x7f1aa00703d0 2024-02-19T22:12:59.016 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.015+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1aa010e610 con 0x7f1aa00703d0 2024-02-19T22:12:59.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.019+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1a90031430 con 0x7f1aa00703d0 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.019+0000 7f1a977fe700 1 -- 172.21.15.45:0/919047349 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+489 (secure 0 0 0) 0x7f1a900315a0 con 0x7f1aa00703d0 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [global] 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: fsid = f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.45:3300,v1:172.21.15.45:6789] 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [mgr] 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T22:12:59.024 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: [osd] 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 msgr2=0x7f1aa010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 --2- 172.21.15.45:0/919047349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1a90006c70 tx=0x7f1a90016000).stop 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 shutdown_connections 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 --2- 172.21.15.45:0/919047349 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1aa00703d0 0x7f1aa010c580 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 >> 172.21.15.45:0/919047349 conn(0x7f1aa00f6520 msgr2=0x7f1aa00f8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 shutdown_connections 2024-02-19T22:12:59.025 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.021+0000 7f1aa51d5700 1 -- 172.21.15.45:0/919047349 wait complete. 2024-02-19T22:12:59.366 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: audit 2024-02-19T22:12:59.016763+0000 mon.smithi045 (mon.0) 8 : audit [INF] from='client.? 172.21.15.45:0/919047349' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T22:12:59.366 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:12:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: audit 2024-02-19T22:12:59.020118+0000 mon.smithi045 (mon.0) 9 : audit [INF] from='client.? 172.21.15.45:0/919047349' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T22:12:59.670 INFO:teuthology.orchestra.run.smithi045.stderr:Generating new minimal ceph.conf... 2024-02-19T22:12:59.670 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z -v /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/mon.smithi045:/var/lib/ceph/mon/ceph-smithi045:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi045/config 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc9bda700 1 Processor -- start 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc9bda700 1 -- start start 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc9bda700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc9bda700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40fef40 con 0x7fbfc4070430 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55542/0 (socket says 172.21.15.45:55542) 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.264+0000 7fbfc37fe700 1 -- 172.21.15.45:0/2335350608 learned_addr learned my addr 172.21.15.45:0/2335350608 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.265+0000 7fbfc37fe700 1 -- 172.21.15.45:0/2335350608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc40ff080 con 0x7fbfc4070430 2024-02-19T22:13:00.276 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.265+0000 7fbfc37fe700 1 --2- 172.21.15.45:0/2335350608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbfb400b5f0 tx=0x7fbfb4006ee0).ready entity=mon.0 client_cookie=234ccf8e826a9233 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.265+0000 7fbfc27fc700 1 -- 172.21.15.45:0/2335350608 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb400d600 con 0x7fbfc4070430 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.265+0000 7fbfc27fc700 1 -- 172.21.15.45:0/2335350608 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbfb400dbf0 con 0x7fbfc4070430 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.265+0000 7fbfc27fc700 1 -- 172.21.15.45:0/2335350608 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb4011cb0 con 0x7fbfc4070430 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2335350608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 msgr2=0x7fbfc40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 --2- 172.21.15.45:0/2335350608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fbfb400b5f0 tx=0x7fbfb4006ee0).stop 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2335350608 shutdown_connections 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 --2- 172.21.15.45:0/2335350608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fea50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2335350608 >> 172.21.15.45:0/2335350608 conn(0x7fbfc40f6530 msgr2=0x7fbfc40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2335350608 shutdown_connections 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2335350608 wait complete. 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.266+0000 7fbfc9bda700 1 Processor -- start 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc9bda700 1 -- start start 2024-02-19T22:13:00.277 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc9bda700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc9bda700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc40fa950 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc37fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55552/0 (socket says 172.21.15.45:55552) 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc37fe700 1 -- 172.21.15.45:0/2233712689 learned_addr learned my addr 172.21.15.45:0/2233712689 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc37fe700 1 -- 172.21.15.45:0/2233712689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb400b200 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc37fe700 1 --2- 172.21.15.45:0/2233712689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbfb4006800 tx=0x7fbfb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb401b570 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbfb401bb60 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc40faaf0 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfb401a9f0 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.267+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc40faf80 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.268+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fbfb400f070 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.268+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fbfb4025cb0 con 0x7fbfc4070430 2024-02-19T22:13:00.278 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.269+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc4045ad0 con 0x7fbfc4070430 2024-02-19T22:13:00.279 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.271+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fbfb4016070 con 0x7fbfc4070430 2024-02-19T22:13:00.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.339+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fbfc4104af0 con 0x7fbfc4070430 2024-02-19T22:13:00.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.339+0000 7fbfbbfff700 1 -- 172.21.15.45:0/2233712689 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fbfb401aea0 con 0x7fbfc4070430 2024-02-19T22:13:00.341 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 msgr2=0x7fbfc40fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 --2- 172.21.15.45:0/2233712689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fbfb4006800 tx=0x7fbfb4006ee0).stop 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 shutdown_connections 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 --2- 172.21.15.45:0/2233712689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbfc4070430 0x7fbfc40fc4d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 >> 172.21.15.45:0/2233712689 conn(0x7fbfc40f6530 msgr2=0x7fbfc40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 shutdown_connections 2024-02-19T22:13:00.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.340+0000 7fbfc9bda700 1 -- 172.21.15.45:0/2233712689 wait complete. 2024-02-19T22:13:00.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: audit 2024-02-19T22:13:00.340184+0000 mon.smithi045 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.45:0/2233712689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:13:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:Restarting the monitor... 2024-02-19T22:13:01.110 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl restart ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:13:01.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:13:01.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: debug 2024-02-19T22:13:01.240+0000 7f60923b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:13:01.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[37680]: debug 2024-02-19T22:13:01.240+0000 7f60923b1700 -1 mon.smithi045@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:13:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 podman[38004]: ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045 2024-02-19T22:13:02.372 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 systemd[1]: ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service: Succeeded. 2024-02-19T22:13:02.372 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 systemd[1]: Stopped Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:13:02.373 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:01 smithi045 systemd[1]: Starting Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:13:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T22:13:02.550 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.501+0000 7fda9644d700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.501+0000 7fda9644d700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.501+0000 7fda9644d700 0 pidfile_write: ignore empty --pid-file 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.505+0000 7fda9644d700 0 load: jerasure load: lrc load: isa 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:13:02.809 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: DB SUMMARY 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi045/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi045/store.db: 000006.log size: 98700 ; 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:13:02.810 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.env: 0x555b0ee161c0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.info_log: 0x555b10525960 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:13:02.811 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi045/store.db 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:13:02.812 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.write_buffer_manager: 0x555b1119a030 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:13:02.813 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:13:02.814 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:13:02.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:13:02.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:13:02.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:13:02.815 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.row_cache: None 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:13:02.816 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:13:02.817 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000005 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:13:02.818 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.merge_operator: 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555b1043dcb0) 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cache_index_and_filter_blocks: 1 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: pin_top_level_index_and_filter: 1 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: index_type: 0 2024-02-19T22:13:02.819 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: data_block_index_type: 0 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: index_shortening: 1 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: hash_index_allow_collision: 1 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: checksum: 1 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: no_block_cache: 0 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_cache: 0x555b10472f10 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_cache_name: BinnedLRUCache 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_cache_options: 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: capacity : 536870912 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: num_shard_bits : 4 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: strict_capacity_limit : 0 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: high_pri_pool_ratio: 0.000 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_cache_compressed: (nil) 2024-02-19T22:13:02.820 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: persistent_cache: (nil) 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_size: 4096 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_size_deviation: 10 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_restart_interval: 16 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: index_block_restart_interval: 1 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: metadata_block_size: 4096 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: partition_filters: 0 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: use_delta_encoding: 1 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: whole_key_filtering: 1 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: verify_compression: 0 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: read_amp_bytes_per_bit: 0 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: format_version: 2 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: enable_index_compression: 1 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: block_align: 0 2024-02-19T22:13:02.821 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:13:02.822 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:13:02.823 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:13:02.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:13:02.825 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:13:02.826 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.506+0000 7fda9644d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.507+0000 7fda9644d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi045/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.507+0000 7fda9644d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.507+0000 7fda9644d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380782508645, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.507+0000 7fda9644d700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.508+0000 7fda9644d700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.508+0000 7fda9644d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380782509597, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94274, "table_properties": {"data_size": 92362, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85393, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708380782, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.508+0000 7fda9644d700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.509+0000 7fda9644d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380782510049, "job": 1, "event": "recovery_finished"} 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.509+0000 7fda9644d700 4 rocksdb: DB pointer 0x555b10537800 2024-02-19T22:13:02.827 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.510+0000 7fda7d265700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.510+0000 7fda7d265700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** DB Stats ** 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** Compaction Stats [default] ** 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:13:02.828 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** Compaction Stats [default] ** 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:13:02.829 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Cumulative compaction: 0.00 GB write, 28.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Interval compaction: 0.00 GB write, 28.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** Compaction Stats [default] ** 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: L0 2/0 93.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Sum 2/0 93.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** Compaction Stats [default] ** 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:13:02.830 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.5 0.00 0.00 1 0.001 0 0 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Cumulative compaction: 0.00 GB write, 28.52 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:13:02.831 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.510+0000 7fda9644d700 0 starting mon.smithi045 rank 0 at public addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] at bind addrs [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi045 fsid f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 1 mon.smithi045@-1(???) e1 preinit fsid f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).mds e1 new map 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).mds e1 print_map 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: e1 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: legacy client fscid: -1 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: No filesystems configured 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T22:13:02.832 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.511+0000 7fda9644d700 0 mon.smithi045@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:00.340184+0000 mon.smithi045 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.45:0/2233712689' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:13:02.512+0000 7fda9644d700 1 mon.smithi045@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 bash[38078]: fd1ea763962e94adc630d166298863b131472105276bc416928d2a3571687abc 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 systemd[1]: Started Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:02.528800+0000 mon.smithi045 (mon.0) 1 : cluster [INF] mon.smithi045 is new leader, mons smithi045 in quorum (ranks 0) 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:02.528846+0000 mon.smithi045 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi045=[v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0]} 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:02.528892+0000 mon.smithi045 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:02.528945+0000 mon.smithi045 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:13:02.833 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:02.529284+0000 mon.smithi045 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:13:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f91206700 1 Processor -- start 2024-02-19T22:13:03.125 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f91206700 1 -- start start 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f91206700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f91206700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c0ff030 con 0x7f6f8c070460 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f8ad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f8ad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55554/0 (socket says 172.21.15.45:55554) 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.123+0000 7f6f8ad9d700 1 -- 172.21.15.45:0/2384859902 learned_addr learned my addr 172.21.15.45:0/2384859902 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.124+0000 7f6f8ad9d700 1 -- 172.21.15.45:0/2384859902 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c0ff170 con 0x7f6f8c070460 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.124+0000 7f6f8ad9d700 1 --2- 172.21.15.45:0/2384859902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6f7c00b5f0 tx=0x7f6f7c006ee0).ready entity=mon.0 client_cookie=19a20d9455cb465f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.124+0000 7f6f89d9b700 1 -- 172.21.15.45:0/2384859902 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7c00d600 con 0x7f6f8c070460 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.124+0000 7f6f89d9b700 1 -- 172.21.15.45:0/2384859902 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6f7c00dbf0 con 0x7f6f8c070460 2024-02-19T22:13:03.126 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.124+0000 7f6f89d9b700 1 -- 172.21.15.45:0/2384859902 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7c011ce0 con 0x7f6f8c070460 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 -- 172.21.15.45:0/2384859902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 msgr2=0x7f6f8c0feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 --2- 172.21.15.45:0/2384859902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f6f7c00b5f0 tx=0x7f6f7c006ee0).stop 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 -- 172.21.15.45:0/2384859902 shutdown_connections 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 --2- 172.21.15.45:0/2384859902 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c0feb40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 -- 172.21.15.45:0/2384859902 >> 172.21.15.45:0/2384859902 conn(0x7f6f8c0f65c0 msgr2=0x7f6f8c0f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 -- 172.21.15.45:0/2384859902 shutdown_connections 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 -- 172.21.15.45:0/2384859902 wait complete. 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.125+0000 7f6f91206700 1 Processor -- start 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f91206700 1 -- start start 2024-02-19T22:13:03.129 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f91206700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f91206700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c10b0c0 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f8ad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f8ad9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55560/0 (socket says 172.21.15.45:55560) 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f8ad9d700 1 -- 172.21.15.45:0/3162229105 learned_addr learned my addr 172.21.15.45:0/3162229105 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f8ad9d700 1 -- 172.21.15.45:0/3162229105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c00b200 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f8ad9d700 1 --2- 172.21.15.45:0/3162229105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6f7c00df90 tx=0x7f6f7c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7c01b570 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6f7c01bb60 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.126+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f7c01ab80 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.127+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c10b2c0 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.127+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c10b750 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.127+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f6f7c00f070 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.127+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f6f7c01bcd0 con 0x7f6f8c070460 2024-02-19T22:13:03.130 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.127+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c045ad0 con 0x7f6f8c070460 2024-02-19T22:13:03.131 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.130+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f6f7c016070 con 0x7f6f8c070460 2024-02-19T22:13:03.198 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.197+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f6f8c104c50 con 0x7f6f8c070460 2024-02-19T22:13:03.200 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.199+0000 7f6f737fe700 1 -- 172.21.15.45:0/3162229105 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6f7c01a370 con 0x7f6f8c070460 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 msgr2=0x7f6f8c10abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 --2- 172.21.15.45:0/3162229105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f6f7c00df90 tx=0x7f6f7c006ee0).stop 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 shutdown_connections 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 --2- 172.21.15.45:0/3162229105 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f8c070460 0x7f6f8c10abd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 >> 172.21.15.45:0/3162229105 conn(0x7f6f8c0f65c0 msgr2=0x7f6f8c0f7fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 shutdown_connections 2024-02-19T22:13:03.202 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.200+0000 7f6f91206700 1 -- 172.21.15.45:0/3162229105 wait complete. 2024-02-19T22:13:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T22:13:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:13:03.848 INFO:teuthology.orchestra.run.smithi045.stderr:Creating mgr... 2024-02-19T22:13:03.849 INFO:teuthology.orchestra.run.smithi045.stderr:Verifying port 9283 ... 2024-02-19T22:13:03.851 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl daemon-reload 2024-02-19T22:13:03.953 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo 2024-02-19T22:13:03.988 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl reset-failed ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo 2024-02-19T22:13:03.994 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Failed to reset failed state of unit ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo.service: Unit ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo.service not loaded. 2024-02-19T22:13:03.994 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl enable ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo 2024-02-19T22:13:04.000 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715.target.wants/ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo.service → /etc/systemd/system/ceph-f6020aec-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:13:04.090 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mgr.smithi045.bvlamo 2024-02-19T22:13:04.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:03.200290+0000 mon.smithi045 (mon.0) 6 : audit [INF] from='client.? 172.21.15.45:0/3162229105' entity='client.admin' 2024-02-19T22:13:04.645 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:13:04.653 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2024-02-19T22:13:04.655 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:13:04.661 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:13:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2024-02-19T22:13:04.662 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:13:04.663 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:13:04.663 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:13:04.670 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2024-02-19T22:13:04.671 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:13:04.679 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:13:04.680 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2024-02-19T22:13:04.680 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T22:13:04.680 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr to start... 2024-02-19T22:13:04.680 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr... 2024-02-19T22:13:04.680 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.218+0000 7f7aa1e0b700 1 Processor -- start 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.218+0000 7f7aa1e0b700 1 -- start start 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.218+0000 7f7aa1e0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.218+0000 7f7aa1e0b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c0fed80 con 0x7f7a9c070430 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7aa0e09700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7aa0e09700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55582/0 (socket says 172.21.15.45:55582) 2024-02-19T22:13:05.221 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7aa0e09700 1 -- 172.21.15.45:0/711772172 learned_addr learned my addr 172.21.15.45:0/711772172 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:05.222 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7aa0e09700 1 -- 172.21.15.45:0/711772172 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c0feec0 con 0x7f7a9c070430 2024-02-19T22:13:05.222 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7aa0e09700 1 --2- 172.21.15.45:0/711772172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7a9000b5f0 tx=0x7f7a90006ee0).ready entity=mon.0 client_cookie=a06da8f97d2cf4b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:05.224 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7a9b7fe700 1 -- 172.21.15.45:0/711772172 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9000d600 con 0x7f7a9c070430 2024-02-19T22:13:05.224 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7a9b7fe700 1 -- 172.21.15.45:0/711772172 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7a9000dbf0 con 0x7f7a9c070430 2024-02-19T22:13:05.224 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.219+0000 7f7a9b7fe700 1 -- 172.21.15.45:0/711772172 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a90011cb0 con 0x7f7a9c070430 2024-02-19T22:13:05.224 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/711772172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 msgr2=0x7f7a9c0fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 --2- 172.21.15.45:0/711772172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f7a9000b5f0 tx=0x7f7a90006ee0).stop 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/711772172 shutdown_connections 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 --2- 172.21.15.45:0/711772172 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fe890 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/711772172 >> 172.21.15.45:0/711772172 conn(0x7f7a9c0f6390 msgr2=0x7f7a9c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/711772172 shutdown_connections 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/711772172 wait complete. 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.220+0000 7f7aa1e0b700 1 Processor -- start 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa1e0b700 1 -- start start 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa1e0b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa1e0b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9c0fa790 con 0x7f7a9c070430 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa0e09700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:05.225 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa0e09700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:55592/0 (socket says 172.21.15.45:55592) 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa0e09700 1 -- 172.21.15.45:0/1235859662 learned_addr learned my addr 172.21.15.45:0/1235859662 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa0e09700 1 -- 172.21.15.45:0/1235859662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9000b200 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.221+0000 7f7aa0e09700 1 --2- 172.21.15.45:0/1235859662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a9000df90 tx=0x7f7a90006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9001b410 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c0fa930 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f7a9001ba00 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9001aa30 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c0fadc0 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7a9000f070 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.222+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7a90025c80 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.223+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c045ad0 con 0x7f7a9c070430 2024-02-19T22:13:05.226 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.225+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7a90016070 con 0x7f7a9c070430 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.296+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f7a9c104930 con 0x7f7a9c070430 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.296+0000 7f7a997fa700 1 -- 172.21.15.45:0/1235859662 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f7a90025460 con 0x7f7a9c070430 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f6020aec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:13:05.304 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "smithi045" 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:13:05.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:13:05.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:13:05.307 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:56.984220+0000", 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:05.308 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 msgr2=0x7f7a9c0fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 --2- 172.21.15.45:0/1235859662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a9000df90 tx=0x7f7a90006ee0).stop 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 shutdown_connections 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 --2- 172.21.15.45:0/1235859662 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a9c070430 0x7f7a9c0fc310 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 >> 172.21.15.45:0/1235859662 conn(0x7f7a9c0f6390 msgr2=0x7f7a9c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 shutdown_connections 2024-02-19T22:13:05.309 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.298+0000 7f7aa1e0b700 1 -- 172.21.15.45:0/1235859662 wait complete. 2024-02-19T22:13:05.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:05.297542+0000 mon.smithi045 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.45:0/1235859662' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:13:06.008 INFO:teuthology.orchestra.run.smithi045.stderr:mgr not available, waiting (1/15)... 2024-02-19T22:13:08.009 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.534+0000 7f74dbccc700 1 Processor -- start 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.534+0000 7f74dbccc700 1 -- start start 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.534+0000 7f74dbccc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.534+0000 7f74dbccc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40fc8a0 con 0x7f74d40703d0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d9a68700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d9a68700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58240/0 (socket says 172.21.15.45:58240) 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d9a68700 1 -- 172.21.15.45:0/3981697606 learned_addr learned my addr 172.21.15.45:0/3981697606 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d9a68700 1 -- 172.21.15.45:0/3981697606 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d41013a0 con 0x7f74d40703d0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d9a68700 1 --2- 172.21.15.45:0/3981697606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f74d000b5f0 tx=0x7f74d0006ee0).ready entity=mon.0 client_cookie=7519a8edfade5f28 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d8a66700 1 -- 172.21.15.45:0/3981697606 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d000d600 con 0x7f74d40703d0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d8a66700 1 -- 172.21.15.45:0/3981697606 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f74d000dbf0 con 0x7f74d40703d0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.535+0000 7f74d8a66700 1 -- 172.21.15.45:0/3981697606 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d0011cb0 con 0x7f74d40703d0 2024-02-19T22:13:08.539 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 -- 172.21.15.45:0/3981697606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 msgr2=0x7f74d4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 --2- 172.21.15.45:0/3981697606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f74d000b5f0 tx=0x7f74d0006ee0).stop 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 -- 172.21.15.45:0/3981697606 shutdown_connections 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 --2- 172.21.15.45:0/3981697606 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d4100eb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 -- 172.21.15.45:0/3981697606 >> 172.21.15.45:0/3981697606 conn(0x7f74d40f6530 msgr2=0x7f74d40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 -- 172.21.15.45:0/3981697606 shutdown_connections 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.536+0000 7f74dbccc700 1 -- 172.21.15.45:0/3981697606 wait complete. 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74dbccc700 1 Processor -- start 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74dbccc700 1 -- start start 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74dbccc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74dbccc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d410d2a0 con 0x7f74d40703d0 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74d9a68700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74d9a68700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58256/0 (socket says 172.21.15.45:58256) 2024-02-19T22:13:08.540 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74d9a68700 1 -- 172.21.15.45:0/901899953 learned_addr learned my addr 172.21.15.45:0/901899953 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.537+0000 7f74d9a68700 1 -- 172.21.15.45:0/901899953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d000b200 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74d9a68700 1 --2- 172.21.15.45:0/901899953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f74d0006a20 tx=0x7f74d0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d001b570 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f74d001bb60 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d410d4a0 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74d001aa20 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d410e0e0 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f74d000f070 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.538+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f74d0025980 con 0x7f74d40703d0 2024-02-19T22:13:08.542 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.539+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d410d930 con 0x7f74d40703d0 2024-02-19T22:13:08.543 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.541+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f74d0016070 con 0x7f74d40703d0 2024-02-19T22:13:08.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.614+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f74d41020c0 con 0x7f74d40703d0 2024-02-19T22:13:08.622 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.614+0000 7f74c27fc700 1 -- 172.21.15.45:0/901899953 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f74d001b6d0 con 0x7f74d40703d0 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f6020aec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:08.623 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "smithi045" 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:13:08.624 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.625 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:56.984220+0000", 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.615+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 msgr2=0x7f74d410cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.615+0000 7f74dbccc700 1 --2- 172.21.15.45:0/901899953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f74d0006a20 tx=0x7f74d0006ee0).stop 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.618+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 shutdown_connections 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.618+0000 7f74dbccc700 1 --2- 172.21.15.45:0/901899953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f74d40703d0 0x7f74d410cdb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.627 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.618+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 >> 172.21.15.45:0/901899953 conn(0x7f74d40f6530 msgr2=0x7f74d40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.618+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 shutdown_connections 2024-02-19T22:13:08.628 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.618+0000 7f74dbccc700 1 -- 172.21.15.45:0/901899953 wait complete. 2024-02-19T22:13:09.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:08.615296+0000 mon.smithi045 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.45:0/901899953' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:13:10.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:09.692684+0000 mon.smithi045 (mon.0) 9 : cluster [INF] Activating manager daemon smithi045.bvlamo 2024-02-19T22:13:10.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:09.695215+0000 mon.smithi045 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi045.bvlamo(active, starting, since 0.0026067s) 2024-02-19T22:13:10.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.695989+0000 mon.smithi045 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:13:10.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.696146+0000 mon.smithi045 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:13:10.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.696278+0000 mon.smithi045 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.696419+0000 mon.smithi045 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.696911+0000 mon.smithi045 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mgr metadata", "who": "smithi045.bvlamo", "id": "smithi045.bvlamo"}]: dispatch 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:09.698747+0000 mon.smithi045 (mon.0) 16 : cluster [INF] Manager daemon smithi045.bvlamo is now available 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.706450+0000 mon.smithi045 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.711300+0000 mon.smithi045 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.712473+0000 mon.smithi045 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.716385+0000 mon.smithi045 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:09.718174+0000 mon.smithi045 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.45:0/59954988' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:12.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:10.698563+0000 mon.smithi045 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi045.bvlamo(active, since 1.00595s) 2024-02-19T22:13:13.424 INFO:teuthology.orchestra.run.smithi045.stderr:mgr not available, waiting (2/15)... 2024-02-19T22:13:14.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:12.698575+0000 mon.smithi045 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi045.bvlamo(active, since 3s) 2024-02-19T22:13:15.427 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:13:19.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:19.343213+0000 mon.smithi045 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.45:0/1002826929' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:13:19.814 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f333a61a700 1 Processor -- start 2024-02-19T22:13:19.814 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f333a61a700 1 -- start start 2024-02-19T22:13:19.814 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f333a61a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f333a61a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33340fc750 con 0x7f33340fbe40 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f3333fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f3333fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58690/0 (socket says 172.21.15.45:58690) 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f3333fff700 1 -- 172.21.15.45:0/3952764840 learned_addr learned my addr 172.21.15.45:0/3952764840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f3333fff700 1 -- 172.21.15.45:0/3952764840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33340fc890 con 0x7f33340fbe40 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.114+0000 7f3333fff700 1 --2- 172.21.15.45:0/3952764840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=93a03148a588bb6d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f3332ffd700 1 -- 172.21.15.45:0/3952764840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332400d600 con 0x7f33340fbe40 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f3332ffd700 1 -- 172.21.15.45:0/3952764840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f332400dbf0 con 0x7f33340fbe40 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f3332ffd700 1 -- 172.21.15.45:0/3952764840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3324011cb0 con 0x7f33340fbe40 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 -- 172.21.15.45:0/3952764840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 msgr2=0x7f33340fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 --2- 172.21.15.45:0/3952764840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f332400b5f0 tx=0x7f3324006ee0).stop 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 -- 172.21.15.45:0/3952764840 shutdown_connections 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 --2- 172.21.15.45:0/3952764840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f33340fc260 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 -- 172.21.15.45:0/3952764840 >> 172.21.15.45:0/3952764840 conn(0x7f33340f7640 msgr2=0x7f33340f9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 -- 172.21.15.45:0/3952764840 shutdown_connections 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.115+0000 7f333a61a700 1 -- 172.21.15.45:0/3952764840 wait complete. 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f333a61a700 1 Processor -- start 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f333a61a700 1 -- start start 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f333a61a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f333a61a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333410b090 con 0x7f33340fbe40 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f3333fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f3333fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58694/0 (socket says 172.21.15.45:58694) 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f3333fff700 1 -- 172.21.15.45:0/1002826929 learned_addr learned my addr 172.21.15.45:0/1002826929 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.116+0000 7f3333fff700 1 -- 172.21.15.45:0/1002826929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332400b200 con 0x7f33340fbe40 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f3333fff700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f332400bf40 tx=0x7f3324006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401b570 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f332401bb60 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333410b290 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f332401aa20 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.117+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333410b720 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f332400f070 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f3330ff9700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 0x7f33200322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3334045ad0 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f33240412e0 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f33337fe700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 0x7f33200322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.118+0000 7f33337fe700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 0x7f33200322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f33280096a0 tx=0x7f3328006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.123+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3324019c90 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.342+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f33340fdc10 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.342+0000 7f3330ff9700 1 -- 172.21.15.45:0/1002826929 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f3324019ab0 con 0x7f33340fbe40 2024-02-19T22:13:19.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 msgr2=0x7f33200322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 0x7f33200322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f33280096a0 tx=0x7f3328006b40).stop 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 msgr2=0x7f333410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f332400bf40 tx=0x7f3324006ee0).stop 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 shutdown_connections 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f332002fe10 0x7f33200322d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 --2- 172.21.15.45:0/1002826929 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33340fbe40 0x7f333410aba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 >> 172.21.15.45:0/1002826929 conn(0x7f33340f7640 msgr2=0x7f33340f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 shutdown_connections 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.344+0000 7f333a61a700 1 -- 172.21.15.45:0/1002826929 wait complete. 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsid": "f6020aec-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:13:19.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 0 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "smithi045" 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "quorum_age": 16, 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:13:19.828 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:13:19.829 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:13:19.830 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ], 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:56.984220+0000", 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: }, 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:13:19.831 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:13:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:mgr is available 2024-02-19T22:13:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling cephadm module... 2024-02-19T22:13:20.880 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T22:13:21.437 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.433+0000 7f4d92560700 1 Processor -- start 2024-02-19T22:13:21.437 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.433+0000 7f4d92560700 1 -- start start 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.433+0000 7f4d92560700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.433+0000 7f4d92560700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c0fb6a0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.433+0000 7f4d8bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58700/0 (socket says 172.21.15.45:58700) 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8bfff700 1 -- 172.21.15.45:0/2497460865 learned_addr learned my addr 172.21.15.45:0/2497460865 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8bfff700 1 -- 172.21.15.45:0/2497460865 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c0fb7e0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8bfff700 1 --2- 172.21.15.45:0/2497460865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4d7c00b870 tx=0x7f4d7c006f70).ready entity=mon.0 client_cookie=c0fe90718d832bc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8affd700 1 -- 172.21.15.45:0/2497460865 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c00da20 con 0x7f4d8c0fad90 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8affd700 1 -- 172.21.15.45:0/2497460865 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d7c010470 con 0x7f4d8c0fad90 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.434+0000 7f4d8affd700 1 -- 172.21.15.45:0/2497460865 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c00f5b0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 -- 172.21.15.45:0/2497460865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 msgr2=0x7f4d8c0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 --2- 172.21.15.45:0/2497460865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4d7c00b870 tx=0x7f4d7c006f70).stop 2024-02-19T22:13:21.438 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 -- 172.21.15.45:0/2497460865 shutdown_connections 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 --2- 172.21.15.45:0/2497460865 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c0fb1b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 -- 172.21.15.45:0/2497460865 >> 172.21.15.45:0/2497460865 conn(0x7f4d8c0f6530 msgr2=0x7f4d8c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 -- 172.21.15.45:0/2497460865 shutdown_connections 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 -- 172.21.15.45:0/2497460865 wait complete. 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.435+0000 7f4d92560700 1 Processor -- start 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d92560700 1 -- start start 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d92560700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d92560700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c108df0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d8bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d8bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58702/0 (socket says 172.21.15.45:58702) 2024-02-19T22:13:21.439 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d8bfff700 1 -- 172.21.15.45:0/3222401258 learned_addr learned my addr 172.21.15.45:0/3222401258 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d8bfff700 1 -- 172.21.15.45:0/3222401258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c00b480 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.436+0000 7f4d8bfff700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4d7c006800 tx=0x7f4d7c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c00f3d0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c108ff0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d7c00fd00 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c019d00 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c109480 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.437+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7f4d7c016030 con 0x7f4d8c0fad90 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.438+0000 7f4d88ff9700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 0x7f4d78032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.441 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.438+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4d7c043ec0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.438+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c045ad0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.438+0000 7f4d8b7fe700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 0x7f4d78032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.442 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.438+0000 7f4d8b7fe700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 0x7f4d78032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4d800096a0 tx=0x7f4d80006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.443 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.442+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d7c014020 con 0x7f4d8c0fad90 2024-02-19T22:13:21.642 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.640+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f4d8c10a4b0 con 0x7f4d8c0fad90 2024-02-19T22:13:21.694 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.693+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4d7c019550 con 0x7f4d8c0fad90 2024-02-19T22:13:21.695 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.694+0000 7f4d88ff9700 1 -- 172.21.15.45:0/3222401258 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4d7c024420 con 0x7f4d8c0fad90 2024-02-19T22:13:21.698 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 msgr2=0x7f4d78032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.698 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 0x7f4d78032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f4d800096a0 tx=0x7f4d80006b40).stop 2024-02-19T22:13:21.698 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 msgr2=0x7f4d8c108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f4d7c006800 tx=0x7f4d7c006f70).stop 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 shutdown_connections 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f4d7802fdc0 0x7f4d78032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 --2- 172.21.15.45:0/3222401258 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d8c0fad90 0x7f4d8c108900 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.696+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 >> 172.21.15.45:0/3222401258 conn(0x7f4d8c0f6530 msgr2=0x7f4d8c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.697+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 shutdown_connections 2024-02-19T22:13:21.699 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.697+0000 7f4d92560700 1 -- 172.21.15.45:0/3222401258 wait complete. 2024-02-19T22:13:22.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:21.642052+0000 mon.smithi045 (mon.0) 25 : audit [INF] from='client.? 172.21.15.45:0/3222401258' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T22:13:22.436 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.047+0000 7f285f3da700 1 Processor -- start 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.047+0000 7f285f3da700 1 -- start start 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285f3da700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285f3da700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28580fb6a0 con 0x7f28580fad90 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285d176700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285d176700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58740/0 (socket says 172.21.15.45:58740) 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285d176700 1 -- 172.21.15.45:0/454852035 learned_addr learned my addr 172.21.15.45:0/454852035 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285d176700 1 -- 172.21.15.45:0/454852035 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28580fb7e0 con 0x7f28580fad90 2024-02-19T22:13:23.050 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f285d176700 1 --2- 172.21.15.45:0/454852035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f285400b5f0 tx=0x7f2854006ee0).ready entity=mon.0 client_cookie=6ffa3aa86a5ae950 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:23.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f2847fff700 1 -- 172.21.15.45:0/454852035 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f285400d600 con 0x7f28580fad90 2024-02-19T22:13:23.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f2847fff700 1 -- 172.21.15.45:0/454852035 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f285400dbf0 con 0x7f28580fad90 2024-02-19T22:13:23.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.048+0000 7f2847fff700 1 -- 172.21.15.45:0/454852035 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2854011cb0 con 0x7f28580fad90 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 -- 172.21.15.45:0/454852035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 msgr2=0x7f28580fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 --2- 172.21.15.45:0/454852035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f285400b5f0 tx=0x7f2854006ee0).stop 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 -- 172.21.15.45:0/454852035 shutdown_connections 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 --2- 172.21.15.45:0/454852035 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f28580fb1b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 -- 172.21.15.45:0/454852035 >> 172.21.15.45:0/454852035 conn(0x7f28580f6530 msgr2=0x7f28580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 -- 172.21.15.45:0/454852035 shutdown_connections 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.049+0000 7f285f3da700 1 -- 172.21.15.45:0/454852035 wait complete. 2024-02-19T22:13:23.053 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285f3da700 1 Processor -- start 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285f3da700 1 -- start start 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285f3da700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285f3da700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285810b040 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285d176700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285d176700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58750/0 (socket says 172.21.15.45:58750) 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285d176700 1 -- 172.21.15.45:0/1362304244 learned_addr learned my addr 172.21.15.45:0/1362304244 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.050+0000 7f285d176700 1 -- 172.21.15.45:0/1362304244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285400b200 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f285d176700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2854002f40 tx=0x7f2854006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f285401b410 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285810b240 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f285401ba00 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f285401a900 con 0x7f28580fad90 2024-02-19T22:13:23.054 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.051+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f285810b6d0 con 0x7f28580fad90 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f285400f070 con 0x7f28580fad90 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f2845ffb700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 0x7f28400322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2854041250 con 0x7f28580fad90 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2858045ad0 con 0x7f28580fad90 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f285c975700 1 -- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 msgr2=0x7f28400322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:23.055 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.052+0000 7f285c975700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 0x7f28400322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:23.058 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.057+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2854016070 con 0x7f28580fad90 2024-02-19T22:13:23.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:21.695246+0000 mon.smithi045 (mon.0) 26 : audit [INF] from='client.? 172.21.15.45:0/3222401258' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T22:13:23.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:21.695292+0000 mon.smithi045 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi045.bvlamo(active, since 12s) 2024-02-19T22:13:23.254 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.253+0000 7f285c975700 1 -- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 msgr2=0x7f28400322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:23.254 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.253+0000 7f285c975700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 0x7f28400322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:13:23.256 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.255+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2858104740 con 0x7f28580fad90 2024-02-19T22:13:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.255+0000 7f2845ffb700 1 -- 172.21.15.45:0/1362304244 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f2854022d50 con 0x7f28580fad90 2024-02-19T22:13:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:13:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T22:13:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:13:23.257 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "active_name": "smithi045.bvlamo", 2024-02-19T22:13:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:13:23.258 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:13:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 msgr2=0x7f28400322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:23.259 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 0x7f28400322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 msgr2=0x7f285810ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f2854002f40 tx=0x7f2854006ee0).stop 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 shutdown_connections 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7f284002fe10 0x7f28400322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 --2- 172.21.15.45:0/1362304244 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28580fad90 0x7f285810ab50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.257+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 >> 172.21.15.45:0/1362304244 conn(0x7f28580f6530 msgr2=0x7f2858070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.258+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 shutdown_connections 2024-02-19T22:13:23.260 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:23.258+0000 7f285f3da700 1 -- 172.21.15.45:0/1362304244 wait complete. 2024-02-19T22:13:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:23.256714+0000 mon.smithi045 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.45:0/1362304244' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:13:23.935 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for the mgr to restart... 2024-02-19T22:13:23.935 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr epoch 5... 2024-02-19T22:13:23.936 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:13:24.517 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.514+0000 7ff1fc51e700 1 Processor -- start 2024-02-19T22:13:24.517 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.514+0000 7ff1fc51e700 1 -- start start 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.514+0000 7ff1fc51e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.514+0000 7ff1fc51e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f40fef40 con 0x7ff1f4070430 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1fa2ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1fa2ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58764/0 (socket says 172.21.15.45:58764) 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1fa2ba700 1 -- 172.21.15.45:0/1459570338 learned_addr learned my addr 172.21.15.45:0/1459570338 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1fa2ba700 1 -- 172.21.15.45:0/1459570338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f40ff080 con 0x7ff1f4070430 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1fa2ba700 1 --2- 172.21.15.45:0/1459570338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff1f000b5f0 tx=0x7ff1f0006ee0).ready entity=mon.0 client_cookie=a50439e05cd91134 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1f92b8700 1 -- 172.21.15.45:0/1459570338 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f000d600 con 0x7ff1f4070430 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1f92b8700 1 -- 172.21.15.45:0/1459570338 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff1f000dbf0 con 0x7ff1f4070430 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.515+0000 7ff1f92b8700 1 -- 172.21.15.45:0/1459570338 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f0011cb0 con 0x7ff1f4070430 2024-02-19T22:13:24.518 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 -- 172.21.15.45:0/1459570338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 msgr2=0x7ff1f40fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/1459570338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff1f000b5f0 tx=0x7ff1f0006ee0).stop 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 -- 172.21.15.45:0/1459570338 shutdown_connections 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/1459570338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fea50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 -- 172.21.15.45:0/1459570338 >> 172.21.15.45:0/1459570338 conn(0x7ff1f40f6530 msgr2=0x7ff1f40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 -- 172.21.15.45:0/1459570338 shutdown_connections 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.516+0000 7ff1fc51e700 1 -- 172.21.15.45:0/1459570338 wait complete. 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fc51e700 1 Processor -- start 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fc51e700 1 -- start start 2024-02-19T22:13:24.521 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fc51e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fc51e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f40fa950 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fa2ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fa2ba700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58768/0 (socket says 172.21.15.45:58768) 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.517+0000 7ff1fa2ba700 1 -- 172.21.15.45:0/523959346 learned_addr learned my addr 172.21.15.45:0/523959346 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1fa2ba700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f000b200 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1fa2ba700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff1f000df90 tx=0x7ff1f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f001b410 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f40faaf0 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff1f001ba00 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1f001aa30 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.518+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f40faf80 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.519+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff1f000f070 con 0x7ff1f4070430 2024-02-19T22:13:24.522 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.519+0000 7ff1e2ffd700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:24.523 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.519+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1dc032990 con 0x7ff1dc02fe10 2024-02-19T22:13:24.523 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.519+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff1f0041140 con 0x7ff1f4070430 2024-02-19T22:13:24.523 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.520+0000 7ff1f9ab9700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 msgr2=0x7ff1dc0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:24.523 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.520+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:24.722 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.721+0000 7ff1f9ab9700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 msgr2=0x7ff1dc0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:24.723 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:24.721+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:13:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:25.122+0000 7ff1f9ab9700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 msgr2=0x7ff1dc0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:25.123 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:25.122+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:13:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:25.923+0000 7ff1f9ab9700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 msgr2=0x7ff1dc0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/1028724476 2024-02-19T22:13:25.928 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:25.923+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:13:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:26.910+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ff1f001ab90 con 0x7ff1f4070430 2024-02-19T22:13:26.912 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:26.910+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 msgr2=0x7ff1dc0322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:26.913 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:26.910+0000 7ff1e2ffd700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:27.915 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.912+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ff1f001b570 con 0x7ff1f4070430 2024-02-19T22:13:27.915 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.912+0000 7ff1e2ffd700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 0x7ff1dc035d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:27.915 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.912+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1dc032990 con 0x7ff1dc033930 2024-02-19T22:13:27.915 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.913+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 0x7ff1dc035d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:27.916 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.913+0000 7ff1f9ab9700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 0x7ff1dc035d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1e40036d0 tx=0x7ff1e4006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:27.916 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.914+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff1dc032990 con 0x7ff1dc033930 2024-02-19T22:13:27.922 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.920+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff1f4059100 con 0x7ff1dc033930 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1e2ffd700 1 -- 172.21.15.45:0/523959346 <== mgr.14114 v2:172.21.15.45:6800/62557027 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff1f4059100 con 0x7ff1dc033930 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 msgr2=0x7ff1dc035d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 0x7ff1dc035d20 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff1e40036d0 tx=0x7ff1e4006ca0).stop 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 msgr2=0x7ff1f40fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff1f000df90 tx=0x7ff1f0006ee0).stop 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 shutdown_connections 2024-02-19T22:13:27.924 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7ff1dc033930 0x7ff1dc035d20 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff1f4070430 0x7ff1f40fc4d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 --2- 172.21.15.45:0/523959346 >> [v2:172.21.15.45:6800/1028724476,v1:172.21.15.45:6801/1028724476] conn(0x7ff1dc02fe10 0x7ff1dc0322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 >> 172.21.15.45:0/523959346 conn(0x7ff1f40f6530 msgr2=0x7ff1f40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 shutdown_connections 2024-02-19T22:13:27.925 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:27.921+0000 7ff1fc51e700 1 -- 172.21.15.45:0/523959346 wait complete. 2024-02-19T22:13:28.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:26.859757+0000 mon.smithi045 (mon.0) 29 : cluster [INF] Active manager daemon smithi045.bvlamo restarted 2024-02-19T22:13:28.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:26.860190+0000 mon.smithi045 (mon.0) 30 : cluster [INF] Activating manager daemon smithi045.bvlamo 2024-02-19T22:13:28.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:26.861785+0000 mon.smithi045 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T22:13:28.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:26.912564+0000 mon.smithi045 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi045.bvlamo(active, starting, since 0.0524468s) 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.913729+0000 mon.smithi045 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.914278+0000 mon.smithi045 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mgr metadata", "who": "smithi045.bvlamo", "id": "smithi045.bvlamo"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.915067+0000 mon.smithi045 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.915219+0000 mon.smithi045 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.915357+0000 mon.smithi045 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:26.917171+0000 mon.smithi045 (mon.0) 38 : cluster [INF] Manager daemon smithi045.bvlamo is now available 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.930562+0000 mon.smithi045 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.933242+0000 mon.smithi045 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:26.935683+0000 mon.smithi045 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.029210+0000 mon.smithi045 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.029906+0000 mon.smithi045 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.031605+0000 mon.smithi045 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.033130+0000 mon.smithi045 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:28.142 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.043520+0000 mon.smithi045 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:28.142 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.044697+0000 mon.smithi045 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:mgr epoch 5 is available 2024-02-19T22:13:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:Setting orchestrator backend to cephadm... 2024-02-19T22:13:28.534 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T22:13:29.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.914916+0000 mgr.smithi045.bvlamo (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:13:29.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:27.915850+0000 mon.smithi045 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi045.bvlamo(active, since 1.05573s) 2024-02-19T22:13:29.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:27.921923+0000 mgr.smithi045.bvlamo (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:13:29.171 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f958213d700 1 Processor -- start 2024-02-19T22:13:29.174 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f958213d700 1 -- start start 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f958213d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f958213d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c0fef80 con 0x7f957c0703f0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58310/0 (socket says 172.21.15.45:58310) 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.170+0000 7f957b7fe700 1 -- 172.21.15.45:0/1343963217 learned_addr learned my addr 172.21.15.45:0/1343963217 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f957b7fe700 1 -- 172.21.15.45:0/1343963217 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c0ff0c0 con 0x7f957c0703f0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f957b7fe700 1 --2- 172.21.15.45:0/1343963217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f956c00b5f0 tx=0x7f956c006ee0).ready entity=mon.0 client_cookie=5d2ef207756d32d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f957a7fc700 1 -- 172.21.15.45:0/1343963217 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956c00d600 con 0x7f957c0703f0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f957a7fc700 1 -- 172.21.15.45:0/1343963217 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f956c00dbf0 con 0x7f957c0703f0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f957a7fc700 1 -- 172.21.15.45:0/1343963217 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956c011ce0 con 0x7f957c0703f0 2024-02-19T22:13:29.175 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f958213d700 1 -- 172.21.15.45:0/1343963217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 msgr2=0x7f957c0fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f958213d700 1 --2- 172.21.15.45:0/1343963217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f956c00b5f0 tx=0x7f956c006ee0).stop 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f958213d700 1 -- 172.21.15.45:0/1343963217 shutdown_connections 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f958213d700 1 --2- 172.21.15.45:0/1343963217 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c0fea90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.171+0000 7f958213d700 1 -- 172.21.15.45:0/1343963217 >> 172.21.15.45:0/1343963217 conn(0x7f957c0f6530 msgr2=0x7f957c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.172+0000 7f958213d700 1 -- 172.21.15.45:0/1343963217 shutdown_connections 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.172+0000 7f958213d700 1 -- 172.21.15.45:0/1343963217 wait complete. 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.172+0000 7f958213d700 1 Processor -- start 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.172+0000 7f958213d700 1 -- start start 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f958213d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f958213d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c10b010 con 0x7f957c0703f0 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f957b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58326/0 (socket says 172.21.15.45:58326) 2024-02-19T22:13:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f957b7fe700 1 -- 172.21.15.45:0/1667927296 learned_addr learned my addr 172.21.15.45:0/1667927296 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f957b7fe700 1 -- 172.21.15.45:0/1667927296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c00b200 con 0x7f957c0703f0 2024-02-19T22:13:29.177 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f957b7fe700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f956c011ed0 tx=0x7f956c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956c01b570 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c10b210 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f956c01bb60 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f956c01aad0 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.173+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c10b6a0 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.174+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f956c00f070 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.174+0000 7f9563fff700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 0x7f95680322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.175+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f956c041370 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.175+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c045ad0 con 0x7f957c0703f0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.175+0000 7f957affd700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 0x7f95680322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:29.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.175+0000 7f957affd700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 0x7f95680322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f95700096a0 tx=0x7f9570006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:29.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.179+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f956c016070 con 0x7f957c0703f0 2024-02-19T22:13:29.374 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.372+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f957c10bfc0 con 0x7f956802fe10 2024-02-19T22:13:29.379 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.377+0000 7f9563fff700 1 -- 172.21.15.45:0/1667927296 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f957c10bfc0 con 0x7f956802fe10 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.379+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 msgr2=0x7f95680322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.379+0000 7f958213d700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 0x7f95680322d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f95700096a0 tx=0x7f9570006b40).stop 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.379+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 msgr2=0x7f957c10ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.379+0000 7f958213d700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f956c011ed0 tx=0x7f956c006ee0).stop 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 shutdown_connections 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f956802fe10 0x7f95680322d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 --2- 172.21.15.45:0/1667927296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f957c0703f0 0x7f957c10ab20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 >> 172.21.15.45:0/1667927296 conn(0x7f957c0f6530 msgr2=0x7f957c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 shutdown_connections 2024-02-19T22:13:29.382 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:29.380+0000 7f958213d700 1 -- 172.21.15.45:0/1667927296 wait complete. 2024-02-19T22:13:30.124 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T22:13:30.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:28.919508+0000 mon.smithi045 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi045.bvlamo(active, since 2s) 2024-02-19T22:13:30.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:29.374264+0000 mgr.smithi045.bvlamo (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:30.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:29.378251+0000 mon.smithi045 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:30.127 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:29.381606+0000 mon.smithi045 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.755+0000 7fc3fe2fd700 1 Processor -- start 2024-02-19T22:13:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.755+0000 7fc3fe2fd700 1 -- start start 2024-02-19T22:13:30.758 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.756+0000 7fc3fe2fd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.756+0000 7fc3fe2fd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80fef60 con 0x7fc3f8070410 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.756+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.756+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58334/0 (socket says 172.21.15.45:58334) 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.756+0000 7fc3f7fff700 1 -- 172.21.15.45:0/1953140576 learned_addr learned my addr 172.21.15.45:0/1953140576 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3f7fff700 1 -- 172.21.15.45:0/1953140576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80ff0a0 con 0x7fc3f8070410 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3f7fff700 1 --2- 172.21.15.45:0/1953140576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc3e800b5f0 tx=0x7fc3e8006ee0).ready entity=mon.0 client_cookie=3373092e3cf42b04 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3f6ffd700 1 -- 172.21.15.45:0/1953140576 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e800d600 con 0x7fc3f8070410 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3f6ffd700 1 -- 172.21.15.45:0/1953140576 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc3e800dbf0 con 0x7fc3f8070410 2024-02-19T22:13:30.759 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3f6ffd700 1 -- 172.21.15.45:0/1953140576 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e8011cb0 con 0x7fc3f8070410 2024-02-19T22:13:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/1953140576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 msgr2=0x7fc3f80fea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/1953140576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc3e800b5f0 tx=0x7fc3e8006ee0).stop 2024-02-19T22:13:30.761 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/1953140576 shutdown_connections 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.757+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/1953140576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fea70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.758+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/1953140576 >> 172.21.15.45:0/1953140576 conn(0x7fc3f80f6530 msgr2=0x7fc3f80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.758+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/1953140576 shutdown_connections 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.758+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/1953140576 wait complete. 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.758+0000 7fc3fe2fd700 1 Processor -- start 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.758+0000 7fc3fe2fd700 1 -- start start 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3fe2fd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3fe2fd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80fa970 con 0x7fc3f8070410 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58340/0 (socket says 172.21.15.45:58340) 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f7fff700 1 -- 172.21.15.45:0/4099576860 learned_addr learned my addr 172.21.15.45:0/4099576860 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f7fff700 1 -- 172.21.15.45:0/4099576860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e800b200 con 0x7fc3f8070410 2024-02-19T22:13:30.762 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f7fff700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc3e800df90 tx=0x7fc3e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e801b410 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fc3e801ba00 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e801aa30 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f80fab10 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.759+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f80fafa0 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.760+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc3e800f070 con 0x7fc3f8070410 2024-02-19T22:13:30.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.760+0000 7fc3f4ff9700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 0x7fc3e40322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.760+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc3e8041180 con 0x7fc3f8070410 2024-02-19T22:13:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.760+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f8045ad0 con 0x7fc3f8070410 2024-02-19T22:13:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.760+0000 7fc3f77fe700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 0x7fc3e40322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:30.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.761+0000 7fc3f77fe700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 0x7fc3e40322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc3ec0096a0 tx=0x7fc3ec006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:30.766 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.765+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3e8016070 con 0x7fc3f8070410 2024-02-19T22:13:30.926 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.924+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3f80008d0 con 0x7fc3e402fe10 2024-02-19T22:13:30.926 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.925+0000 7fc3f4ff9700 1 -- 172.21.15.45:0/4099576860 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fc3f80008d0 con 0x7fc3e402fe10 2024-02-19T22:13:30.926 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: value unchanged 2024-02-19T22:13:30.928 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 msgr2=0x7fc3e40322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 0x7fc3e40322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fc3ec0096a0 tx=0x7fc3ec006b40).stop 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 msgr2=0x7fc3f80fc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc3e800df90 tx=0x7fc3e8006ee0).stop 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 shutdown_connections 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fc3e402fe10 0x7fc3e40322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 --2- 172.21.15.45:0/4099576860 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc3f8070410 0x7fc3f80fc4f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.926+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 >> 172.21.15.45:0/4099576860 conn(0x7fc3f80f6530 msgr2=0x7fc3f80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.927+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 shutdown_connections 2024-02-19T22:13:30.929 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:30.927+0000 7fc3fe2fd700 1 -- 172.21.15.45:0/4099576860 wait complete. 2024-02-19T22:13:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:Generating ssh key... 2024-02-19T22:13:31.514 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T22:13:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.039+0000 7f5399a23700 1 Processor -- start 2024-02-19T22:13:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.039+0000 7f5399a23700 1 -- start start 2024-02-19T22:13:32.043 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5399a23700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5399a23700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5394072920 con 0x7f5394072500 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5392ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5392ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58346/0 (socket says 172.21.15.45:58346) 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5392ffd700 1 -- 172.21.15.45:0/1797434798 learned_addr learned my addr 172.21.15.45:0/1797434798 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5392ffd700 1 -- 172.21.15.45:0/1797434798 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394070f90 con 0x7f5394072500 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5392ffd700 1 --2- 172.21.15.45:0/1797434798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f538400b5f0 tx=0x7f5384006ee0).ready entity=mon.0 client_cookie=29740c4bddd71fca server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5391ffb700 1 -- 172.21.15.45:0/1797434798 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538400d600 con 0x7f5394072500 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5391ffb700 1 -- 172.21.15.45:0/1797434798 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f538400dbf0 con 0x7f5394072500 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.040+0000 7f5391ffb700 1 -- 172.21.15.45:0/1797434798 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5384011cb0 con 0x7f5394072500 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.041+0000 7f5399a23700 1 -- 172.21.15.45:0/1797434798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 msgr2=0x7f5394070aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.041+0000 7f5399a23700 1 --2- 172.21.15.45:0/1797434798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f538400b5f0 tx=0x7f5384006ee0).stop 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.042+0000 7f5399a23700 1 -- 172.21.15.45:0/1797434798 shutdown_connections 2024-02-19T22:13:32.044 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.042+0000 7f5399a23700 1 --2- 172.21.15.45:0/1797434798 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f5394070aa0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:32.045 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.042+0000 7f5399a23700 1 -- 172.21.15.45:0/1797434798 >> 172.21.15.45:0/1797434798 conn(0x7f53940f76a0 msgr2=0x7f53940f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:32.045 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.042+0000 7f5399a23700 1 -- 172.21.15.45:0/1797434798 shutdown_connections 2024-02-19T22:13:32.045 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.042+0000 7f5399a23700 1 -- 172.21.15.45:0/1797434798 wait complete. 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5399a23700 1 Processor -- start 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5399a23700 1 -- start start 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5399a23700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5399a23700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539410d3a0 con 0x7f5394072500 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5392ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5392ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58352/0 (socket says 172.21.15.45:58352) 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.043+0000 7f5392ffd700 1 -- 172.21.15.45:0/287166952 learned_addr learned my addr 172.21.15.45:0/287166952 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f5392ffd700 1 -- 172.21.15.45:0/287166952 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538400b200 con 0x7f5394072500 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f5392ffd700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f538400df90 tx=0x7f5384006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:32.047 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538401b410 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539410d5a0 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f538401ba00 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f538401aa30 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.044+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539410e150 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.045+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f538400f070 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.045+0000 7f537b7fe700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 0x7f53800322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.045+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5384041330 con 0x7f5394072500 2024-02-19T22:13:32.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.045+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5394059100 con 0x7f5394072500 2024-02-19T22:13:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.045+0000 7f53927fc700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 0x7f53800322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:32.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.046+0000 7f53927fc700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 0x7f53800322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f53880096a0 tx=0x7f5388006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:32.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.050+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5384016030 con 0x7f5394072500 2024-02-19T22:13:32.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:30.925772+0000 mgr.smithi045.bvlamo (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:32.211 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.209+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5394102210 con 0x7f538002fe10 2024-02-19T22:13:32.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.324+0000 7f537b7fe700 1 -- 172.21.15.45:0/287166952 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5394102210 con 0x7f538002fe10 2024-02-19T22:13:32.328 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 msgr2=0x7f53800322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 0x7f53800322d0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f53880096a0 tx=0x7f5388006b40).stop 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 msgr2=0x7f539410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f538400df90 tx=0x7f5384006ee0).stop 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 shutdown_connections 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f538002fe10 0x7f53800322d0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 --2- 172.21.15.45:0/287166952 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5394072500 0x7f539410ceb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 >> 172.21.15.45:0/287166952 conn(0x7f53940f76a0 msgr2=0x7f53940f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 shutdown_connections 2024-02-19T22:13:32.329 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:32.326+0000 7f5399a23700 1 -- 172.21.15.45:0/287166952 wait complete. 2024-02-19T22:13:32.996 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.565+0000 7f56b75a7700 1 Processor -- start 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.565+0000 7f56b75a7700 1 -- start start 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.565+0000 7f56b75a7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.565+0000 7f56b75a7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b00fc880 con 0x7f56b00703f0 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f56b5343700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f56b5343700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58368/0 (socket says 172.21.15.45:58368) 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f56b5343700 1 -- 172.21.15.45:0/1947628216 learned_addr learned my addr 172.21.15.45:0/1947628216 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:33.569 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f56b5343700 1 -- 172.21.15.45:0/1947628216 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b0101380 con 0x7f56b00703f0 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f56b5343700 1 --2- 172.21.15.45:0/1947628216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f56a000b5f0 tx=0x7f56a0006ee0).ready entity=mon.0 client_cookie=5a25087c020d4ae8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f569ffff700 1 -- 172.21.15.45:0/1947628216 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a000d600 con 0x7f56b00703f0 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f569ffff700 1 -- 172.21.15.45:0/1947628216 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f56a000dbf0 con 0x7f56b00703f0 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.566+0000 7f569ffff700 1 -- 172.21.15.45:0/1947628216 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a0011ce0 con 0x7f56b00703f0 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 -- 172.21.15.45:0/1947628216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 msgr2=0x7f56b0100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 --2- 172.21.15.45:0/1947628216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f56a000b5f0 tx=0x7f56a0006ee0).stop 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 -- 172.21.15.45:0/1947628216 shutdown_connections 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 --2- 172.21.15.45:0/1947628216 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b0100e90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 -- 172.21.15.45:0/1947628216 >> 172.21.15.45:0/1947628216 conn(0x7f56b00f6530 msgr2=0x7f56b00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 -- 172.21.15.45:0/1947628216 shutdown_connections 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.567+0000 7f56b75a7700 1 -- 172.21.15.45:0/1947628216 wait complete. 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b75a7700 1 Processor -- start 2024-02-19T22:13:33.570 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b75a7700 1 -- start start 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b75a7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b75a7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56b010d290 con 0x7f56b00703f0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b5343700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b5343700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58382/0 (socket says 172.21.15.45:58382) 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b5343700 1 -- 172.21.15.45:0/2337199354 learned_addr learned my addr 172.21.15.45:0/2337199354 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b5343700 1 -- 172.21.15.45:0/2337199354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a000b200 con 0x7f56b00703f0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.568+0000 7f56b5343700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f56a0011ed0 tx=0x7f56a0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.569+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a001b570 con 0x7f56b00703f0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.569+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f56a001bb60 con 0x7f56b00703f0 2024-02-19T22:13:33.572 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.569+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b010d490 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.569+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56a001aad0 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.569+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56b010e0d0 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f56a000f070 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f569dffb700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 0x7f56980322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56b010d920 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f56a0041370 con 0x7f56b00703f0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f56b4b42700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 0x7f56980322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:33.573 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.570+0000 7f56b4b42700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 0x7f56980322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:33.575 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.574+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56a0006a70 con 0x7f56b00703f0 2024-02-19T22:13:33.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:32.211373+0000 mgr.smithi045.bvlamo (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T22:13:33.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:32.323188+0000 mon.smithi045 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:33.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:32.324981+0000 mon.smithi045 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:33.736 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.734+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f56b010e3d0 con 0x7f569802fe10 2024-02-19T22:13:33.736 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.735+0000 7f569dffb700 1 -- 172.21.15.45:0/2337199354 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f56b010e3d0 con 0x7f569802fe10 2024-02-19T22:13:33.736 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqUbJ4tb5VYHd7111Rm7h4gbbhYiYlorB4Nz8uikTBRx6REcJs1gW/PiWsNhnbo2QNNRawM7JJS0Wz/srPDjZdFbCWAycoepA13P+GQ8K0Q92pxwhgkr80ln+jbTLAUNDpa0ymoBs9tQaUor49u5bMyZVvDIR4EOKvgY2tUdp0L+kxjL9YyOjNAxWgq0el7lQl3oo6P55C1QcZVI6lAX8X+HoVw50r53a3J5JDEf9vAc3f3H50HEkwSje1F/sPfaoWs1MTI1RRWxvVizXZ3bCjK/R3VPXdwX0NR4DlxS9h7gDwbqBw8MP7Fbq+0WNzf2tE9TKtrV3umt7CLJ/pEftDvqNAoVUZT+GP00dwlNtIVLlOLDFlGLJ4/9k/fSSLkgowJ2cmLJ+S6PrJA5cikXyIzqSN58A8vxnmqz5/xIuTYSt2ZGn2yadUnXLKh5W2KffI2Cjesa7zFbJ/0u/D/xnGCBhbv6ljyFu3EQVxBYVDrzx7lSKjVaGDYSDgZN2mjeU= ceph-f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 msgr2=0x7f56980322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 0x7f56980322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f56ac0096a0 tx=0x7f56ac006b40).stop 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 msgr2=0x7f56b010cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f56a0011ed0 tx=0x7f56a0006ee0).stop 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 shutdown_connections 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f569802fe10 0x7f56980322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 --2- 172.21.15.45:0/2337199354 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56b00703f0 0x7f56b010cda0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 >> 172.21.15.45:0/2337199354 conn(0x7f56b00f6530 msgr2=0x7f56b00f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 shutdown_connections 2024-02-19T22:13:33.739 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:33.737+0000 7f56b75a7700 1 -- 172.21.15.45:0/2337199354 wait complete. 2024-02-19T22:13:34.446 INFO:teuthology.orchestra.run.smithi045.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:13:34.446 INFO:teuthology.orchestra.run.smithi045.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T22:13:34.447 INFO:teuthology.orchestra.run.smithi045.stderr:Adding host smithi045... 2024-02-19T22:13:34.447 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi045 172.21.15.45 2024-02-19T22:13:34.983 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.961+0000 7f1dc9999700 1 Processor -- start 2024-02-19T22:13:34.983 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.961+0000 7f1dc9999700 1 -- start start 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc9999700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc9999700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc40fc7b0 con 0x7f1dc40fbea0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58386/0 (socket says 172.21.15.45:58386) 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc2ffd700 1 -- 172.21.15.45:0/1374667806 learned_addr learned my addr 172.21.15.45:0/1374667806 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc2ffd700 1 -- 172.21.15.45:0/1374667806 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40fc8f0 con 0x7f1dc40fbea0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc2ffd700 1 --2- 172.21.15.45:0/1374667806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1dac00b5f0 tx=0x7f1dac006ee0).ready entity=mon.0 client_cookie=892412b41a01dc3f server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.962+0000 7f1dc1ffb700 1 -- 172.21.15.45:0/1374667806 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac00d600 con 0x7f1dc40fbea0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc1ffb700 1 -- 172.21.15.45:0/1374667806 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1dac00dbf0 con 0x7f1dc40fbea0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc1ffb700 1 -- 172.21.15.45:0/1374667806 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac011cb0 con 0x7f1dc40fbea0 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 -- 172.21.15.45:0/1374667806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 msgr2=0x7f1dc40fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 --2- 172.21.15.45:0/1374667806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1dac00b5f0 tx=0x7f1dac006ee0).stop 2024-02-19T22:13:34.984 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 -- 172.21.15.45:0/1374667806 shutdown_connections 2024-02-19T22:13:34.985 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 --2- 172.21.15.45:0/1374667806 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40fc2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:34.985 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 -- 172.21.15.45:0/1374667806 >> 172.21.15.45:0/1374667806 conn(0x7f1dc40f76a0 msgr2=0x7f1dc40f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:34.985 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 -- 172.21.15.45:0/1374667806 shutdown_connections 2024-02-19T22:13:34.985 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.963+0000 7f1dc9999700 1 -- 172.21.15.45:0/1374667806 wait complete. 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc9999700 1 Processor -- start 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc9999700 1 -- start start 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc9999700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc9999700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc4070a60 con 0x7f1dc40fbea0 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:58400/0 (socket says 172.21.15.45:58400) 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc2ffd700 1 -- 172.21.15.45:0/2570437993 learned_addr learned my addr 172.21.15.45:0/2570437993 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.964+0000 7f1dc2ffd700 1 -- 172.21.15.45:0/2570437993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dac00b200 con 0x7f1dc40fbea0 2024-02-19T22:13:34.986 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dc2ffd700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1dac00bf40 tx=0x7f1dac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac01b570 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f1dac01bb60 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc4070c00 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dac01aa20 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4071090 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.965+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f1dac00f070 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.966+0000 7f1dbb7fe700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 0x7f1da4032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.966+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f1dac0414b0 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.966+0000 7f1dc27fc700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 0x7f1da4032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.966+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc4059100 con 0x7f1dc40fbea0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.966+0000 7f1dc27fc700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 0x7f1da4032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f1db40096a0 tx=0x7f1db4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:34.987 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:34.970+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dac016030 con 0x7f1dc40fbea0 2024-02-19T22:13:35.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:33.736098+0000 mgr.smithi045.bvlamo (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:35.152 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.151+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}) v1 -- 0x7f1dc4104b80 con 0x7f1da402fe70 2024-02-19T22:13:35.760 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Added host 'smithi045' 2024-02-19T22:13:35.761 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.758+0000 7f1dbb7fe700 1 -- 172.21.15.45:0/2570437993 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f1dc4104b80 con 0x7f1da402fe70 2024-02-19T22:13:35.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 msgr2=0x7f1da4032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:35.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 0x7f1da4032330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f1db40096a0 tx=0x7f1db4006b40).stop 2024-02-19T22:13:35.763 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 msgr2=0x7f1dc40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f1dac00bf40 tx=0x7f1dac006ee0).stop 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 shutdown_connections 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f1da402fe70 0x7f1da4032330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 --2- 172.21.15.45:0/2570437993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1dc40fbea0 0x7f1dc40725e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 >> 172.21.15.45:0/2570437993 conn(0x7f1dc40f76a0 msgr2=0x7f1dc40f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 shutdown_connections 2024-02-19T22:13:35.764 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:35.761+0000 7f1dc9999700 1 -- 172.21.15.45:0/2570437993 wait complete. 2024-02-19T22:13:36.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:35.153000+0000 mgr.smithi045.bvlamo (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi045", "addr": "172.21.15.45", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:36.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:35.758761+0000 mon.smithi045 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:36.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:35.759354+0000 mon.smithi045 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:36.428 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying mon service with default placement... 2024-02-19T22:13:36.428 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.000+0000 7fa9526d3700 1 Processor -- start 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.000+0000 7fa9526d3700 1 -- start start 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa9526d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa9526d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0fef40 con 0x7fa94c070430 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45942/0 (socket says 172.21.15.45:45942) 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94bfff700 1 -- 172.21.15.45:0/3553820876 learned_addr learned my addr 172.21.15.45:0/3553820876 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94bfff700 1 -- 172.21.15.45:0/3553820876 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0ff080 con 0x7fa94c070430 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94bfff700 1 --2- 172.21.15.45:0/3553820876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa93c00b5f0 tx=0x7fa93c006ee0).ready entity=mon.0 client_cookie=620ec5a5676a3d85 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.005 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94affd700 1 -- 172.21.15.45:0/3553820876 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c00d600 con 0x7fa94c070430 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94affd700 1 -- 172.21.15.45:0/3553820876 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa93c00dbf0 con 0x7fa94c070430 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.001+0000 7fa94affd700 1 -- 172.21.15.45:0/3553820876 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c011cb0 con 0x7fa94c070430 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 -- 172.21.15.45:0/3553820876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 msgr2=0x7fa94c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 --2- 172.21.15.45:0/3553820876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa93c00b5f0 tx=0x7fa93c006ee0).stop 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 -- 172.21.15.45:0/3553820876 shutdown_connections 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 --2- 172.21.15.45:0/3553820876 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c0fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 -- 172.21.15.45:0/3553820876 >> 172.21.15.45:0/3553820876 conn(0x7fa94c0f6530 msgr2=0x7fa94c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 -- 172.21.15.45:0/3553820876 shutdown_connections 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.002+0000 7fa9526d3700 1 -- 172.21.15.45:0/3553820876 wait complete. 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa9526d3700 1 Processor -- start 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa9526d3700 1 -- start start 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa9526d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa9526d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c10b030 con 0x7fa94c070430 2024-02-19T22:13:37.006 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa94bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45948/0 (socket says 172.21.15.45:45948) 2024-02-19T22:13:37.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa94bfff700 1 -- 172.21.15.45:0/2026644905 learned_addr learned my addr 172.21.15.45:0/2026644905 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa94bfff700 1 -- 172.21.15.45:0/2026644905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c00b200 con 0x7fa94c070430 2024-02-19T22:13:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.003+0000 7fa94bfff700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa93c006800 tx=0x7fa93c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c01b570 con 0x7fa94c070430 2024-02-19T22:13:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa93c01bb60 con 0x7fa94c070430 2024-02-19T22:13:37.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c10b230 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa93c01a9f0 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c10b6c0 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.004+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa93c00f070 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.005+0000 7fa948ff9700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 0x7fa938032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.005+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa93c042220 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.005+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c045ad0 con 0x7fa94c070430 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.005+0000 7fa94b7fe700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 0x7fa938032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:37.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.005+0000 7fa94b7fe700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 0x7fa938032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa9400096a0 tx=0x7fa940006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:37.010 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.009+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa93c016070 con 0x7fa94c070430 2024-02-19T22:13:37.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:35.759098+0000 mgr.smithi045.bvlamo (mgr.14114) 9 : cephadm [INF] Added host smithi045 2024-02-19T22:13:37.196 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.194+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fa94c10bf20 con 0x7fa93802fe70 2024-02-19T22:13:37.200 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.198+0000 7fa948ff9700 1 -- 172.21.15.45:0/2026644905 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa94c10bf20 con 0x7fa93802fe70 2024-02-19T22:13:37.200 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.200+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 msgr2=0x7fa938032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.200+0000 7fa9526d3700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 0x7fa938032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa9400096a0 tx=0x7fa940006b40).stop 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.200+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 msgr2=0x7fa94c10ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.200+0000 7fa9526d3700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa93c006800 tx=0x7fa93c006ee0).stop 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 shutdown_connections 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7fa93802fe70 0x7fa938032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 --2- 172.21.15.45:0/2026644905 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa94c070430 0x7fa94c10ab40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 >> 172.21.15.45:0/2026644905 conn(0x7fa94c0f6530 msgr2=0x7fa94c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 shutdown_connections 2024-02-19T22:13:37.203 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:37.201+0000 7fa9526d3700 1 -- 172.21.15.45:0/2026644905 wait complete. 2024-02-19T22:13:37.792 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying mgr service with default placement... 2024-02-19T22:13:37.792 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T22:13:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7626d700 1 Processor -- start 2024-02-19T22:13:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7626d700 1 -- start start 2024-02-19T22:13:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7626d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7626d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d700fb440 con 0x7f4d700fab30 2024-02-19T22:13:38.388 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7526b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7526b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45962/0 (socket says 172.21.15.45:45962) 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7526b700 1 -- 172.21.15.45:0/968338223 learned_addr learned my addr 172.21.15.45:0/968338223 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.352+0000 7f4d7526b700 1 -- 172.21.15.45:0/968338223 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d700fb580 con 0x7f4d700fab30 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7526b700 1 --2- 172.21.15.45:0/968338223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4d6c00b5f0 tx=0x7f4d6c006ee0).ready entity=mon.0 client_cookie=c057fec3a0d104cc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d63fff700 1 -- 172.21.15.45:0/968338223 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c00d600 con 0x7f4d700fab30 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d63fff700 1 -- 172.21.15.45:0/968338223 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d6c00dbf0 con 0x7f4d700fab30 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d63fff700 1 -- 172.21.15.45:0/968338223 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c011cb0 con 0x7f4d700fab30 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 -- 172.21.15.45:0/968338223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 msgr2=0x7f4d700faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 --2- 172.21.15.45:0/968338223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4d6c00b5f0 tx=0x7f4d6c006ee0).stop 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 -- 172.21.15.45:0/968338223 shutdown_connections 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 --2- 172.21.15.45:0/968338223 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d700faf50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 -- 172.21.15.45:0/968338223 >> 172.21.15.45:0/968338223 conn(0x7f4d700f6390 msgr2=0x7f4d700f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 -- 172.21.15.45:0/968338223 shutdown_connections 2024-02-19T22:13:38.389 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.353+0000 7f4d7626d700 1 -- 172.21.15.45:0/968338223 wait complete. 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7626d700 1 Processor -- start 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7626d700 1 -- start start 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7626d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7626d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7010ae60 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7526b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7526b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45976/0 (socket says 172.21.15.45:45976) 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.354+0000 7f4d7526b700 1 -- 172.21.15.45:0/966472873 learned_addr learned my addr 172.21.15.45:0/966472873 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d7526b700 1 -- 172.21.15.45:0/966472873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c00b200 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d7526b700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4d6c011ea0 tx=0x7f4d6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c01b570 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4d6c01bb60 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7010b060 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c01a8d0 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.355+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7010b4f0 con 0x7f4d700fab30 2024-02-19T22:13:38.390 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4d6c00f070 con 0x7f4d700fab30 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d61ffb700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 0x7f4d580322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4d6c041450 con 0x7f4d700fab30 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70045ad0 con 0x7f4d700fab30 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d74a6a700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 0x7f4d580322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.356+0000 7f4d74a6a700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 0x7f4d580322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:38.391 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.360+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d6c016070 con 0x7f4d700fab30 2024-02-19T22:13:38.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:37.196405+0000 mgr.smithi045.bvlamo (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:38.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:37.197168+0000 mgr.smithi045.bvlamo (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T22:13:38.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:37.199395+0000 mon.smithi045 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:38.549 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.547+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d7010be10 con 0x7f4d5802fe10 2024-02-19T22:13:38.552 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.551+0000 7f4d61ffb700 1 -- 172.21.15.45:0/966472873 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4d7010be10 con 0x7f4d5802fe10 2024-02-19T22:13:38.553 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 msgr2=0x7f4d580322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 0x7f4d580322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).stop 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 msgr2=0x7f4d7010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f4d6c011ea0 tx=0x7f4d6c006ee0).stop 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 shutdown_connections 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f4d5802fe10 0x7f4d580322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 --2- 172.21.15.45:0/966472873 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4d700fab30 0x7f4d7010a970 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 >> 172.21.15.45:0/966472873 conn(0x7f4d700f6390 msgr2=0x7f4d70070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:38.555 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 shutdown_connections 2024-02-19T22:13:38.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:38.553+0000 7f4d7626d700 1 -- 172.21.15.45:0/966472873 wait complete. 2024-02-19T22:13:39.174 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying crash service with default placement... 2024-02-19T22:13:39.174 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f41093a0700 1 Processor -- start 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f41093a0700 1 -- start start 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f41093a0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f41093a0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040fed80 con 0x7f4104070430 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f4103fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f4103fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45980/0 (socket says 172.21.15.45:45980) 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.812+0000 7f4103fff700 1 -- 172.21.15.45:0/3638671915 learned_addr learned my addr 172.21.15.45:0/3638671915 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f4103fff700 1 -- 172.21.15.45:0/3638671915 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040feec0 con 0x7f4104070430 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f4103fff700 1 --2- 172.21.15.45:0/3638671915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f40f400b5f0 tx=0x7f40f4006ee0).ready entity=mon.0 client_cookie=cb7312a9b1d382ac server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f4102ffd700 1 -- 172.21.15.45:0/3638671915 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40f400d600 con 0x7f4104070430 2024-02-19T22:13:39.816 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f4102ffd700 1 -- 172.21.15.45:0/3638671915 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f40f400dbf0 con 0x7f4104070430 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f4102ffd700 1 -- 172.21.15.45:0/3638671915 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40f4011cb0 con 0x7f4104070430 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f41093a0700 1 -- 172.21.15.45:0/3638671915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 msgr2=0x7f41040fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f41093a0700 1 --2- 172.21.15.45:0/3638671915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f40f400b5f0 tx=0x7f40f4006ee0).stop 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f41093a0700 1 -- 172.21.15.45:0/3638671915 shutdown_connections 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.813+0000 7f41093a0700 1 --2- 172.21.15.45:0/3638671915 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fe890 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.814+0000 7f41093a0700 1 -- 172.21.15.45:0/3638671915 >> 172.21.15.45:0/3638671915 conn(0x7f41040f6390 msgr2=0x7f41040f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.814+0000 7f41093a0700 1 -- 172.21.15.45:0/3638671915 shutdown_connections 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.814+0000 7f41093a0700 1 -- 172.21.15.45:0/3638671915 wait complete. 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.814+0000 7f41093a0700 1 Processor -- start 2024-02-19T22:13:39.817 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.814+0000 7f41093a0700 1 -- start start 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f41093a0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f41093a0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41040fa790 con 0x7f4104070430 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4103fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4103fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:45990/0 (socket says 172.21.15.45:45990) 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4103fff700 1 -- 172.21.15.45:0/2971977819 learned_addr learned my addr 172.21.15.45:0/2971977819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4103fff700 1 -- 172.21.15.45:0/2971977819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f400b200 con 0x7f4104070430 2024-02-19T22:13:39.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4103fff700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f40f400df90 tx=0x7f40f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40f401b410 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f40f401ba00 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41040fa930 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.815+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40f401aa30 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.816+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41040fadc0 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.816+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f40f400f070 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.817+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4104045ad0 con 0x7f4104070430 2024-02-19T22:13:39.820 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.817+0000 7f4100ff9700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 0x7f40f0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.817+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f40f4041140 con 0x7f4104070430 2024-02-19T22:13:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.817+0000 7f41037fe700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 0x7f40f0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.817+0000 7f41037fe700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 0x7f40f0032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f40f80096a0 tx=0x7f40f8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:39.822 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:39.821+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40f401ab90 con 0x7f4104070430 2024-02-19T22:13:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:38.549405+0000 mgr.smithi045.bvlamo (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:38.549907+0000 mgr.smithi045.bvlamo (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T22:13:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:38.552102+0000 mon.smithi045 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:40.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.007+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f41040fb000 con 0x7f40f002fe70 2024-02-19T22:13:40.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.011+0000 7f4100ff9700 1 -- 172.21.15.45:0/2971977819 <== mgr.14114 v2:172.21.15.45:6800/62557027 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f41040fb000 con 0x7f40f002fe70 2024-02-19T22:13:40.012 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 msgr2=0x7f40f0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 0x7f40f0032330 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f40f80096a0 tx=0x7f40f8006b40).stop 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 msgr2=0x7f41040fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f40f400df90 tx=0x7f40f4006ee0).stop 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 shutdown_connections 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f40f002fe70 0x7f40f0032330 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 --2- 172.21.15.45:0/2971977819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4104070430 0x7f41040fc310 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:40.015 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 >> 172.21.15.45:0/2971977819 conn(0x7f41040f6390 msgr2=0x7f41040f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:40.016 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 shutdown_connections 2024-02-19T22:13:40.016 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:40.013+0000 7f41093a0700 1 -- 172.21.15.45:0/2971977819 wait complete. 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling mgr prometheus module... 2024-02-19T22:13:40.746 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T22:13:41.330 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.328+0000 7f5db06a6700 1 Processor -- start 2024-02-19T22:13:41.330 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.328+0000 7f5db06a6700 1 -- start start 2024-02-19T22:13:41.330 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5db06a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:41.331 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5db06a6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da80fef80 con 0x7f5da80703f0 2024-02-19T22:13:41.331 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dae442700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:41.333 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dae442700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46002/0 (socket says 172.21.15.45:46002) 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dae442700 1 -- 172.21.15.45:0/4210119050 learned_addr learned my addr 172.21.15.45:0/4210119050 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dae442700 1 -- 172.21.15.45:0/4210119050 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da80ff0c0 con 0x7f5da80703f0 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dae442700 1 --2- 172.21.15.45:0/4210119050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5da400b5f0 tx=0x7f5da4006ee0).ready entity=mon.0 client_cookie=d1c8ec96752b1d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dad440700 1 -- 172.21.15.45:0/4210119050 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da400d600 con 0x7f5da80703f0 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dad440700 1 -- 172.21.15.45:0/4210119050 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5da400dbf0 con 0x7f5da80703f0 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.329+0000 7f5dad440700 1 -- 172.21.15.45:0/4210119050 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da4011ce0 con 0x7f5da80703f0 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 -- 172.21.15.45:0/4210119050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 msgr2=0x7f5da80fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4210119050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f5da400b5f0 tx=0x7f5da4006ee0).stop 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 -- 172.21.15.45:0/4210119050 shutdown_connections 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4210119050 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da80fea90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 -- 172.21.15.45:0/4210119050 >> 172.21.15.45:0/4210119050 conn(0x7f5da80f6530 msgr2=0x7f5da80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 -- 172.21.15.45:0/4210119050 shutdown_connections 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.330+0000 7f5db06a6700 1 -- 172.21.15.45:0/4210119050 wait complete. 2024-02-19T22:13:41.334 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5db06a6700 1 Processor -- start 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5db06a6700 1 -- start start 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5db06a6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5db06a6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da810b080 con 0x7f5da80703f0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5dae442700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5dae442700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46004/0 (socket says 172.21.15.45:46004) 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5dae442700 1 -- 172.21.15.45:0/4124393829 learned_addr learned my addr 172.21.15.45:0/4124393829 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.331+0000 7f5dae442700 1 -- 172.21.15.45:0/4124393829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da400b200 con 0x7f5da80703f0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5dae442700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5da40115c0 tx=0x7f5da4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da401b410 con 0x7f5da80703f0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da810b280 con 0x7f5da80703f0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5da401ba00 con 0x7f5da80703f0 2024-02-19T22:13:41.335 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da401aaa0 con 0x7f5da80703f0 2024-02-19T22:13:41.336 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.332+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da810b710 con 0x7f5da80703f0 2024-02-19T22:13:41.336 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5da400f070 con 0x7f5da80703f0 2024-02-19T22:13:41.336 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5d9affd700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 0x7f5d900322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:41.336 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5da4041020 con 0x7f5da80703f0 2024-02-19T22:13:41.337 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5dadc41700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 0x7f5d900322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:41.337 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da8045ad0 con 0x7f5da80703f0 2024-02-19T22:13:41.337 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.333+0000 7f5dadc41700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 0x7f5d900322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5d9c0096a0 tx=0x7f5d9c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:41.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.337+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5da4016070 con 0x7f5da80703f0 2024-02-19T22:13:41.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:40.009127+0000 mgr.smithi045.bvlamo (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:41.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:40.009790+0000 mgr.smithi045.bvlamo (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T22:13:41.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:40.011954+0000 mon.smithi045 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.45:0/567225831' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:41.538 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:41.537+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f5da8104b30 con 0x7f5da80703f0 2024-02-19T22:13:42.016 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.014+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f5da4024930 con 0x7f5da80703f0 2024-02-19T22:13:42.017 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.015+0000 7f5d9affd700 1 -- 172.21.15.45:0/4124393829 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f5da40224f0 con 0x7f5da80703f0 2024-02-19T22:13:42.019 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 msgr2=0x7f5d900322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 0x7f5d900322d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5d9c0096a0 tx=0x7f5d9c006b40).stop 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 msgr2=0x7f5da810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f5da40115c0 tx=0x7f5da4006ee0).stop 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 shutdown_connections 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f5d9002fe10 0x7f5d900322d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 --2- 172.21.15.45:0/4124393829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5da80703f0 0x7f5da810ab90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.017+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 >> 172.21.15.45:0/4124393829 conn(0x7f5da80f6530 msgr2=0x7f5da80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:42.020 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.019+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 shutdown_connections 2024-02-19T22:13:42.021 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:42.019+0000 7f5db06a6700 1 -- 172.21.15.45:0/4124393829 wait complete. 2024-02-19T22:13:42.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:41.538297+0000 mon.smithi045 (mon.0) 59 : audit [INF] from='client.? 172.21.15.45:0/4124393829' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T22:13:42.778 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying prometheus service with default placement... 2024-02-19T22:13:42.778 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T22:13:43.032 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:42.016517+0000 mon.smithi045 (mon.0) 60 : audit [INF] from='client.? 172.21.15.45:0/4124393829' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T22:13:43.032 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:42.016568+0000 mon.smithi045 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi045.bvlamo(active, since 15s) 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08ca777700 1 Processor -- start 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08ca777700 1 -- start start 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08ca777700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08ca777700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c40fefa0 con 0x7f08c40703d0 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08c3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08c3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46044/0 (socket says 172.21.15.45:46044) 2024-02-19T22:13:43.414 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.409+0000 7f08c3fff700 1 -- 172.21.15.45:0/1460434152 learned_addr learned my addr 172.21.15.45:0/1460434152 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08c3fff700 1 -- 172.21.15.45:0/1460434152 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c40ff0e0 con 0x7f08c40703d0 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08c3fff700 1 --2- 172.21.15.45:0/1460434152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08b400b5f0 tx=0x7f08b4006ee0).ready entity=mon.0 client_cookie=901fa1e9dc1f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08c2ffd700 1 -- 172.21.15.45:0/1460434152 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b400d600 con 0x7f08c40703d0 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08c2ffd700 1 -- 172.21.15.45:0/1460434152 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f08b400dbf0 con 0x7f08c40703d0 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08c2ffd700 1 -- 172.21.15.45:0/1460434152 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b4011cb0 con 0x7f08c40703d0 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08ca777700 1 -- 172.21.15.45:0/1460434152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 msgr2=0x7f08c40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08ca777700 1 --2- 172.21.15.45:0/1460434152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f08b400b5f0 tx=0x7f08b4006ee0).stop 2024-02-19T22:13:43.415 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.410+0000 7f08ca777700 1 -- 172.21.15.45:0/1460434152 shutdown_connections 2024-02-19T22:13:43.416 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.411+0000 7f08ca777700 1 --2- 172.21.15.45:0/1460434152 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c40feab0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.411+0000 7f08ca777700 1 -- 172.21.15.45:0/1460434152 >> 172.21.15.45:0/1460434152 conn(0x7f08c40f6530 msgr2=0x7f08c40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.411+0000 7f08ca777700 1 -- 172.21.15.45:0/1460434152 shutdown_connections 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.411+0000 7f08ca777700 1 -- 172.21.15.45:0/1460434152 wait complete. 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.411+0000 7f08ca777700 1 Processor -- start 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08ca777700 1 -- start start 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08ca777700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08ca777700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c410b060 con 0x7f08c40703d0 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08c3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08c3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:46048/0 (socket says 172.21.15.45:46048) 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08c3fff700 1 -- 172.21.15.45:0/2552420735 learned_addr learned my addr 172.21.15.45:0/2552420735 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08c3fff700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b400b200 con 0x7f08c40703d0 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.412+0000 7f08c3fff700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f08b4006a20 tx=0x7f08b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:43.417 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b401b570 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c410b260 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f08b401bb60 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08b401aa20 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c410b670 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.413+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f08b400f070 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.414+0000 7f08c0ff9700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.414+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f08b4042340 con 0x7f08c40703d0 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.414+0000 7f08c37fe700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/62557027 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.414+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:43.418 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.414+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c4059100 con 0x7f08c40703d0 2024-02-19T22:13:43.420 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.419+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08b4022d20 con 0x7f08c40703d0 2024-02-19T22:13:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.615+0000 7f08c37fe700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/62557027 2024-02-19T22:13:43.616 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.615+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:13:43.626 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:43.625+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f08c4104c20 con 0x7f08b0004490 2024-02-19T22:13:44.017 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:44.016+0000 7f08c37fe700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/62557027 2024-02-19T22:13:44.017 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:44.016+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:13:44.818 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:44.817+0000 7f08c37fe700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/62557027 2024-02-19T22:13:44.819 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:44.817+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:13:46.421 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:46.418+0000 7f08c37fe700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/62557027 2024-02-19T22:13:46.421 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:46.418+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T22:13:47.296 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:47.295+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f08b401bcd0 con 0x7f08c40703d0 2024-02-19T22:13:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:47.295+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 msgr2=0x7f08b0006950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:47.297 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:47.295+0000 7f08c0ff9700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.296+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f08b4043400 con 0x7f08c40703d0 2024-02-19T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.296+0000 7f08c0ff9700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 0x7f08b003ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.296+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 --> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f08c4104c20 con 0x7f08b0008300 2024-02-19T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.296+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 0x7f08b003ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:48.298 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.297+0000 7f08c37fe700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 0x7f08b003ea10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f08b80036d0 tx=0x7f08b8006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.301+0000 7f08c0ff9700 1 -- 172.21.15.45:0/2552420735 <== mgr.14142 v2:172.21.15.45:6800/172486393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f08c4104c20 con 0x7f08b0008300 2024-02-19T22:13:48.303 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T22:13:48.305 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 msgr2=0x7f08b003ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 0x7f08b003ea10 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f08b80036d0 tx=0x7f08b8006cb0).stop 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 msgr2=0x7f08c410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f08b4006a20 tx=0x7f08b4006ee0).stop 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 shutdown_connections 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f08b0008300 0x7f08b003ea10 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f08c40703d0 0x7f08c410ab70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 --2- 172.21.15.45:0/2552420735 >> [v2:172.21.15.45:6800/62557027,v1:172.21.15.45:6801/62557027] conn(0x7f08b0004490 0x7f08b0006950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 >> 172.21.15.45:0/2552420735 conn(0x7f08c40f6530 msgr2=0x7f08c40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 shutdown_connections 2024-02-19T22:13:48.306 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:48.303+0000 7f08ca777700 1 -- 172.21.15.45:0/2552420735 wait complete. 2024-02-19T22:13:48.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:47.243767+0000 mon.smithi045 (mon.0) 62 : cluster [INF] Active manager daemon smithi045.bvlamo restarted 2024-02-19T22:13:48.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:47.244176+0000 mon.smithi045 (mon.0) 63 : cluster [INF] Activating manager daemon smithi045.bvlamo 2024-02-19T22:13:48.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:47.245864+0000 mon.smithi045 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:47.296669+0000 mon.smithi045 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi045.bvlamo(active, starting, since 0.0525731s) 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.297610+0000 mon.smithi045 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.298125+0000 mon.smithi045 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mgr metadata", "who": "smithi045.bvlamo", "id": "smithi045.bvlamo"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.298744+0000 mon.smithi045 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.298899+0000 mon.smithi045 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.299058+0000 mon.smithi045 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:47.312411+0000 mon.smithi045 (mon.0) 71 : cluster [INF] Manager daemon smithi045.bvlamo is now available 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.415405+0000 mon.smithi045 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.417042+0000 mon.smithi045 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.439859+0000 mon.smithi045 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:48.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:47.443081+0000 mon.smithi045 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying grafana service with default placement... 2024-02-19T22:13:49.002 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.514+0000 7fad04ffa700 1 Processor -- start 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7fad04ffa700 1 -- start start 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7fad04ffa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7fad04ffa700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000fb6a0 con 0x7fad000fad90 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7facfe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7facfe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42310/0 (socket says 172.21.15.45:42310) 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7facfe59c700 1 -- 172.21.15.45:0/2108976078 learned_addr learned my addr 172.21.15.45:0/2108976078 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.515+0000 7facfe59c700 1 -- 172.21.15.45:0/2108976078 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000fb7e0 con 0x7fad000fad90 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7facfe59c700 1 --2- 172.21.15.45:0/2108976078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7facf000b5f0 tx=0x7facf0006ee0).ready entity=mon.0 client_cookie=5fe52a3cfcdcd598 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:49.556 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7facfd59a700 1 -- 172.21.15.45:0/2108976078 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf000d600 con 0x7fad000fad90 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7facfd59a700 1 -- 172.21.15.45:0/2108976078 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7facf000dbf0 con 0x7fad000fad90 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7facfd59a700 1 -- 172.21.15.45:0/2108976078 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf0011cb0 con 0x7fad000fad90 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 -- 172.21.15.45:0/2108976078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 msgr2=0x7fad000fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 --2- 172.21.15.45:0/2108976078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7facf000b5f0 tx=0x7facf0006ee0).stop 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 -- 172.21.15.45:0/2108976078 shutdown_connections 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 --2- 172.21.15.45:0/2108976078 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad000fb1b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 -- 172.21.15.45:0/2108976078 >> 172.21.15.45:0/2108976078 conn(0x7fad000f6530 msgr2=0x7fad000f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 -- 172.21.15.45:0/2108976078 shutdown_connections 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.516+0000 7fad04ffa700 1 -- 172.21.15.45:0/2108976078 wait complete. 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7fad04ffa700 1 Processor -- start 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7fad04ffa700 1 -- start start 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7fad04ffa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7fad04ffa700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad00070470 con 0x7fad000fad90 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7facfe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:49.557 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7facfe59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42326/0 (socket says 172.21.15.45:42326) 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7facfe59c700 1 -- 172.21.15.45:0/3451326166 learned_addr learned my addr 172.21.15.45:0/3451326166 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7facfe59c700 1 -- 172.21.15.45:0/3451326166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf000b200 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.517+0000 7facfe59c700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7facf0011ea0 tx=0x7facf0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf001b570 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7facf001bb60 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad00070670 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facf001a8d0 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0010def0 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.518+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7facf000f070 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.519+0000 7facf6ffd700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 0x7face0032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.519+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7facf0041540 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.519+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad00045ad0 con 0x7fad000fad90 2024-02-19T22:13:49.558 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.519+0000 7facfdd9b700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 0x7face0032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:49.559 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.519+0000 7facfdd9b700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 0x7face0032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:49.559 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.523+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facf0016070 con 0x7fad000fad90 2024-02-19T22:13:49.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:48.298698+0000 mon.smithi045 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi045.bvlamo(active, since 1.0546s) 2024-02-19T22:13:49.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:48.299853+0000 mgr.smithi045.bvlamo (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T22:13:49.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:48.301973+0000 mon.smithi045 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:49.711 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.709+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 --> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fad000fcc50 con 0x7face002ff70 2024-02-19T22:13:49.714 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.713+0000 7facf6ffd700 1 -- 172.21.15.45:0/3451326166 <== mgr.14142 v2:172.21.15.45:6800/172486393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fad000fcc50 con 0x7face002ff70 2024-02-19T22:13:49.714 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 msgr2=0x7face0032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 0x7face0032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).stop 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 msgr2=0x7fad0010d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7facf0011ea0 tx=0x7facf0006ee0).stop 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 shutdown_connections 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7face002ff70 0x7face0032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 --2- 172.21.15.45:0/3451326166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad000fad90 0x7fad0010d840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.715+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 >> 172.21.15.45:0/3451326166 conn(0x7fad000f6530 msgr2=0x7fad000f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.716+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 shutdown_connections 2024-02-19T22:13:49.717 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:49.716+0000 7fad04ffa700 1 -- 172.21.15.45:0/3451326166 wait complete. 2024-02-19T22:13:50.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:49.303075+0000 mon.smithi045 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi045.bvlamo(active, since 2s) 2024-02-19T22:13:50.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:49.711136+0000 mgr.smithi045.bvlamo (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:50.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:49.711651+0000 mgr.smithi045.bvlamo (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T22:13:50.390 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:49.713752+0000 mon.smithi045 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:50.408 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying node-exporter service with default placement... 2024-02-19T22:13:50.408 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T22:13:50.990 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.987+0000 7f4de7fff700 1 Processor -- start 2024-02-19T22:13:50.990 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de7fff700 1 -- start start 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de7fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80fedc0 con 0x7f4de80703f0 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42336/0 (socket says 172.21.15.45:42336) 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de6ffd700 1 -- 172.21.15.45:0/919517155 learned_addr learned my addr 172.21.15.45:0/919517155 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de6ffd700 1 -- 172.21.15.45:0/919517155 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80fef00 con 0x7f4de80703f0 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.988+0000 7f4de6ffd700 1 --2- 172.21.15.45:0/919517155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4dd800b5f0 tx=0x7f4dd8006ee0).ready entity=mon.0 client_cookie=3ad5d6ee6eccec17 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de5ffb700 1 -- 172.21.15.45:0/919517155 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd800d600 con 0x7f4de80703f0 2024-02-19T22:13:50.991 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de5ffb700 1 -- 172.21.15.45:0/919517155 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4dd800dbf0 con 0x7f4de80703f0 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de5ffb700 1 -- 172.21.15.45:0/919517155 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd8011ce0 con 0x7f4de80703f0 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 -- 172.21.15.45:0/919517155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 msgr2=0x7f4de80fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 --2- 172.21.15.45:0/919517155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f4dd800b5f0 tx=0x7f4dd8006ee0).stop 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 -- 172.21.15.45:0/919517155 shutdown_connections 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 --2- 172.21.15.45:0/919517155 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de80fe8d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 -- 172.21.15.45:0/919517155 >> 172.21.15.45:0/919517155 conn(0x7f4de80f6390 msgr2=0x7f4de80f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.989+0000 7f4de7fff700 1 -- 172.21.15.45:0/919517155 shutdown_connections 2024-02-19T22:13:50.993 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de7fff700 1 -- 172.21.15.45:0/919517155 wait complete. 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de7fff700 1 Processor -- start 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de7fff700 1 -- start start 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de7fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de810aec0 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42350/0 (socket says 172.21.15.45:42350) 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.990+0000 7f4de6ffd700 1 -- 172.21.15.45:0/2118561 learned_addr learned my addr 172.21.15.45:0/2118561 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4de6ffd700 1 -- 172.21.15.45:0/2118561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd800b200 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4de6ffd700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4dd80115c0 tx=0x7f4dd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd801b410 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f4dd801ba00 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de810b0c0 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4dd801aaa0 con 0x7f4de80703f0 2024-02-19T22:13:50.994 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.991+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de810b550 con 0x7f4de80703f0 2024-02-19T22:13:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.992+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f4dd800f070 con 0x7f4de80703f0 2024-02-19T22:13:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.992+0000 7f4ddf7fe700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 0x7f4dc8032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.992+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f4dd8041250 con 0x7f4de80703f0 2024-02-19T22:13:50.995 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.992+0000 7f4de67fc700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 0x7f4dc8032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.993+0000 7f4de67fc700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 0x7f4dc8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4dd00096a0 tx=0x7f4dd0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:50.996 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.993+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de8045ad0 con 0x7f4de80703f0 2024-02-19T22:13:50.998 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:50.997+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4dd8010be0 con 0x7f4de80703f0 2024-02-19T22:13:51.188 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.187+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 --> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f4de8104a40 con 0x7f4dc802ff70 2024-02-19T22:13:51.192 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.191+0000 7f4ddf7fe700 1 -- 172.21.15.45:0/2118561 <== mgr.14142 v2:172.21.15.45:6800/172486393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f4de8104a40 con 0x7f4dc802ff70 2024-02-19T22:13:51.192 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 msgr2=0x7f4dc8032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 0x7f4dc8032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f4dd00096a0 tx=0x7f4dd0006b40).stop 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 msgr2=0x7f4de810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4dd80115c0 tx=0x7f4dd8006ee0).stop 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 shutdown_connections 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f4dc802ff70 0x7f4dc8032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 --2- 172.21.15.45:0/2118561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4de80703f0 0x7f4de810a9d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 >> 172.21.15.45:0/2118561 conn(0x7f4de80f6390 msgr2=0x7f4de80f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:51.195 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 shutdown_connections 2024-02-19T22:13:51.196 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:51.193+0000 7f4de7fff700 1 -- 172.21.15.45:0/2118561 wait complete. 2024-02-19T22:13:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:Deploying alertmanager service with default placement... 2024-02-19T22:13:52.107 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T22:13:52.373 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:51.188941+0000 mgr.smithi045.bvlamo (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:52.373 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:51.189578+0000 mgr.smithi045.bvlamo (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T22:13:52.373 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:51.191679+0000 mon.smithi045 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.737+0000 7f0ad333a700 1 Processor -- start 2024-02-19T22:13:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.737+0000 7f0ad333a700 1 -- start start 2024-02-19T22:13:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.737+0000 7f0ad333a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.737+0000 7f0ad333a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc0fb440 con 0x7f0acc0fab30 2024-02-19T22:13:52.741 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad2338700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.744 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad2338700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42356/0 (socket says 172.21.15.45:42356) 2024-02-19T22:13:52.744 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad2338700 1 -- 172.21.15.45:0/3963434852 learned_addr learned my addr 172.21.15.45:0/3963434852 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:52.744 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad2338700 1 -- 172.21.15.45:0/3963434852 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0acc0fb580 con 0x7f0acc0fab30 2024-02-19T22:13:52.744 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad2338700 1 --2- 172.21.15.45:0/3963434852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0ac800b5f0 tx=0x7f0ac8006ee0).ready entity=mon.0 client_cookie=655959d784ceeeff server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad1336700 1 -- 172.21.15.45:0/3963434852 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac800d600 con 0x7f0acc0fab30 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad1336700 1 -- 172.21.15.45:0/3963434852 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0ac800dbf0 con 0x7f0acc0fab30 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.738+0000 7f0ad1336700 1 -- 172.21.15.45:0/3963434852 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac8011cb0 con 0x7f0acc0fab30 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 -- 172.21.15.45:0/3963434852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 msgr2=0x7f0acc0faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 --2- 172.21.15.45:0/3963434852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f0ac800b5f0 tx=0x7f0ac8006ee0).stop 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 -- 172.21.15.45:0/3963434852 shutdown_connections 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 --2- 172.21.15.45:0/3963434852 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc0faf50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 -- 172.21.15.45:0/3963434852 >> 172.21.15.45:0/3963434852 conn(0x7f0acc0f6390 msgr2=0x7f0acc0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 -- 172.21.15.45:0/3963434852 shutdown_connections 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.739+0000 7f0ad333a700 1 -- 172.21.15.45:0/3963434852 wait complete. 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad333a700 1 Processor -- start 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad333a700 1 -- start start 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad333a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.745 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad333a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0acc109da0 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad2338700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad2338700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42368/0 (socket says 172.21.15.45:42368) 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad2338700 1 -- 172.21.15.45:0/2169826083 learned_addr learned my addr 172.21.15.45:0/2169826083 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.740+0000 7f0ad2338700 1 -- 172.21.15.45:0/2169826083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac800b200 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0ad2338700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0ac8002f40 tx=0x7f0ac8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac801b410 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f0ac801ba00 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0acc109fa0 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ac801a900 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.741+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0acc10a430 con 0x7f0acc0fab30 2024-02-19T22:13:52.746 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.742+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0acc045ad0 con 0x7f0acc0fab30 2024-02-19T22:13:52.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.742+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0ac800f070 con 0x7f0acc0fab30 2024-02-19T22:13:52.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.742+0000 7f0abaffd700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 0x7f0ab40323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.742+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0ac8041bc0 con 0x7f0acc0fab30 2024-02-19T22:13:52.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.743+0000 7f0ad1b37700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 0x7f0ab40323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.743+0000 7f0ad1b37700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 0x7f0ab40323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0abc0096a0 tx=0x7f0abc006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.749 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.746+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ac8023d30 con 0x7f0acc0fab30 2024-02-19T22:13:52.936 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.933+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 --> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f0acc10ac90 con 0x7f0ab402ff20 2024-02-19T22:13:52.939 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.937+0000 7f0abaffd700 1 -- 172.21.15.45:0/2169826083 <== mgr.14142 v2:172.21.15.45:6800/172486393 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f0acc10ac90 con 0x7f0ab402ff20 2024-02-19T22:13:52.940 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 msgr2=0x7f0ab40323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 0x7f0ab40323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0abc0096a0 tx=0x7f0abc006b40).stop 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 msgr2=0x7f0acc1098b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f0ac8002f40 tx=0x7f0ac8006ee0).stop 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 shutdown_connections 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f0ab402ff20 0x7f0ab40323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 --2- 172.21.15.45:0/2169826083 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0acc0fab30 0x7f0acc1098b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.942 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 >> 172.21.15.45:0/2169826083 conn(0x7f0acc0f6390 msgr2=0x7f0acc0f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 shutdown_connections 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:52.939+0000 7f0ad333a700 1 -- 172.21.15.45:0/2169826083 wait complete. 2024-02-19T22:13:53.588 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.149+0000 7f5dbe7dc700 1 Processor -- start 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.149+0000 7f5dbe7dc700 1 -- start start 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5dbe7dc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5dbe7dc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db80fc8e0 con 0x7f5db8070390 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42372/0 (socket says 172.21.15.45:42372) 2024-02-19T22:13:54.154 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db7fff700 1 -- 172.21.15.45:0/742982129 learned_addr learned my addr 172.21.15.45:0/742982129 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db7fff700 1 -- 172.21.15.45:0/742982129 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db81013e0 con 0x7f5db8070390 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db7fff700 1 --2- 172.21.15.45:0/742982129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5da800b5f0 tx=0x7f5da8006ee0).ready entity=mon.0 client_cookie=139378721f437caf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db6ffd700 1 -- 172.21.15.45:0/742982129 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da800d600 con 0x7f5db8070390 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db6ffd700 1 -- 172.21.15.45:0/742982129 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5da800dbf0 con 0x7f5db8070390 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.150+0000 7f5db6ffd700 1 -- 172.21.15.45:0/742982129 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da8011cb0 con 0x7f5db8070390 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/742982129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 msgr2=0x7f5db8100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/742982129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5da800b5f0 tx=0x7f5da8006ee0).stop 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/742982129 shutdown_connections 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/742982129 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db8100ef0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/742982129 >> 172.21.15.45:0/742982129 conn(0x7f5db80f6530 msgr2=0x7f5db80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/742982129 shutdown_connections 2024-02-19T22:13:54.155 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.151+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/742982129 wait complete. 2024-02-19T22:13:54.156 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.152+0000 7f5dbe7dc700 1 Processor -- start 2024-02-19T22:13:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.152+0000 7f5dbe7dc700 1 -- start start 2024-02-19T22:13:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.152+0000 7f5dbe7dc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.152+0000 7f5dbe7dc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db810d290 con 0x7f5db8070390 2024-02-19T22:13:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:54.157 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:42374/0 (socket says 172.21.15.45:42374) 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db7fff700 1 -- 172.21.15.45:0/933220826 learned_addr learned my addr 172.21.15.45:0/933220826 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db7fff700 1 -- 172.21.15.45:0/933220826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da800b200 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db7fff700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5da8006800 tx=0x7f5da8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da801b570 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5da801bb60 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db810d490 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5da801a9f0 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.153+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db810e0d0 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.154+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f5da800f070 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.154+0000 7f5db4ff9700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 0x7f5da4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.154+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db810d920 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.154+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5da8041520 con 0x7f5db8070390 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.154+0000 7f5db77fe700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 0x7f5da4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:54.158 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.155+0000 7f5db77fe700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 0x7f5da4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5dac0096a0 tx=0x7f5dac006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:54.161 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.159+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5da8022d70 con 0x7f5db8070390 2024-02-19T22:13:54.320 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.317+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5db8102140 con 0x7f5db8070390 2024-02-19T22:13:54.322 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.320+0000 7f5db4ff9700 1 -- 172.21.15.45:0/933220826 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5da8022310 con 0x7f5db8070390 2024-02-19T22:13:54.325 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 msgr2=0x7f5da4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:54.325 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 0x7f5da4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5dac0096a0 tx=0x7f5dac006b40).stop 2024-02-19T22:13:54.325 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 msgr2=0x7f5db810cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:54.325 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5da8006800 tx=0x7f5da8006ee0).stop 2024-02-19T22:13:54.325 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 shutdown_connections 2024-02-19T22:13:54.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f5da402ff70 0x7f5da4032430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:54.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 --2- 172.21.15.45:0/933220826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5db8070390 0x7f5db810cda0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:54.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 >> 172.21.15.45:0/933220826 conn(0x7f5db80f6530 msgr2=0x7f5db80f7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:54.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 shutdown_connections 2024-02-19T22:13:54.326 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:54.322+0000 7f5dbe7dc700 1 -- 172.21.15.45:0/933220826 wait complete. 2024-02-19T22:13:54.346 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:52.936416+0000 mgr.smithi045.bvlamo (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:54.346 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:13:52.936923+0000 mgr.smithi045.bvlamo (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T22:13:54.346 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:52.939024+0000 mon.smithi045 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:55.230 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T22:13:55.512 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:54.322436+0000 mon.smithi045 (mon.0) 82 : audit [INF] from='client.? 172.21.15.45:0/933220826' entity='client.admin' 2024-02-19T22:13:56.003 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:55.999+0000 7f593bf90700 1 Processor -- start 2024-02-19T22:13:56.003 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:55.999+0000 7f593bf90700 1 -- start start 2024-02-19T22:13:56.003 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f593bf90700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:56.003 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f593bf90700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59340fc8c0 con 0x7f59340703b0 2024-02-19T22:13:56.003 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f5939d2c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f5939d2c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36854/0 (socket says 172.21.15.45:36854) 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f5939d2c700 1 -- 172.21.15.45:0/1326525562 learned_addr learned my addr 172.21.15.45:0/1326525562 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f5939d2c700 1 -- 172.21.15.45:0/1326525562 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59341013c0 con 0x7f59340703b0 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.000+0000 7f5939d2c700 1 --2- 172.21.15.45:0/1326525562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f593000b5f0 tx=0x7f5930006ee0).ready entity=mon.0 client_cookie=43ea49f585934dba server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f5938d2a700 1 -- 172.21.15.45:0/1326525562 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f593000d600 con 0x7f59340703b0 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f5938d2a700 1 -- 172.21.15.45:0/1326525562 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f593000dbf0 con 0x7f59340703b0 2024-02-19T22:13:56.004 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f5938d2a700 1 -- 172.21.15.45:0/1326525562 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5930011cb0 con 0x7f59340703b0 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 -- 172.21.15.45:0/1326525562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 msgr2=0x7f5934100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 --2- 172.21.15.45:0/1326525562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f593000b5f0 tx=0x7f5930006ee0).stop 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 -- 172.21.15.45:0/1326525562 shutdown_connections 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 --2- 172.21.15.45:0/1326525562 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f5934100ed0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 -- 172.21.15.45:0/1326525562 >> 172.21.15.45:0/1326525562 conn(0x7f59340f6530 msgr2=0x7f59340f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 -- 172.21.15.45:0/1326525562 shutdown_connections 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.001+0000 7f593bf90700 1 -- 172.21.15.45:0/1326525562 wait complete. 2024-02-19T22:13:56.007 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f593bf90700 1 Processor -- start 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f593bf90700 1 -- start start 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f593bf90700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f593bf90700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593410d270 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f5939d2c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f5939d2c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36870/0 (socket says 172.21.15.45:36870) 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.002+0000 7f5939d2c700 1 -- 172.21.15.45:0/303494411 learned_addr learned my addr 172.21.15.45:0/303494411 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f5939d2c700 1 -- 172.21.15.45:0/303494411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593000b200 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f5939d2c700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5930006800 tx=0x7f5930006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f593001b570 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593410d470 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f593001bb60 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f593001a9f0 con 0x7f59340703b0 2024-02-19T22:13:56.008 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.003+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593410e080 con 0x7f59340703b0 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f593000f070 con 0x7f59340703b0 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f59227fc700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 0x7f591c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5930041590 con 0x7f59340703b0 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f593952b700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 0x7f591c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5934059100 con 0x7f59340703b0 2024-02-19T22:13:56.009 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.004+0000 7f593952b700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 0x7f591c0323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:56.011 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.009+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5930016020 con 0x7f59340703b0 2024-02-19T22:13:56.170 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.167+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f593410eeb0 con 0x7f59340703b0 2024-02-19T22:13:56.172 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.170+0000 7f59227fc700 1 -- 172.21.15.45:0/303494411 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7f5930019aa0 con 0x7f59340703b0 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 msgr2=0x7f591c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 0x7f591c0323e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).stop 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 msgr2=0x7f593410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f5930006800 tx=0x7f5930006ee0).stop 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 shutdown_connections 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f591c02ff20 0x7f591c0323e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 --2- 172.21.15.45:0/303494411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59340703b0 0x7f593410cd80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 >> 172.21.15.45:0/303494411 conn(0x7f59340f6530 msgr2=0x7f59340f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 shutdown_connections 2024-02-19T22:13:56.176 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:56.172+0000 7f593bf90700 1 -- 172.21.15.45:0/303494411 wait complete. 2024-02-19T22:13:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:Enabling the dashboard module... 2024-02-19T22:13:57.316 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T22:13:57.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:56.172369+0000 mon.smithi045 (mon.0) 83 : audit [INF] from='client.? 172.21.15.45:0/303494411' entity='client.admin' 2024-02-19T22:13:57.318 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:56.916225+0000 mon.smithi045 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:57.900 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f997e96f700 1 Processor -- start 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f997e96f700 1 -- start start 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f997e96f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f997e96f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99780fef40 con 0x7f9978070540 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f9977fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f9977fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36884/0 (socket says 172.21.15.45:36884) 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.897+0000 7f9977fff700 1 -- 172.21.15.45:0/3555641755 learned_addr learned my addr 172.21.15.45:0/3555641755 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:57.901 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.898+0000 7f9977fff700 1 -- 172.21.15.45:0/3555641755 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99780ff080 con 0x7f9978070540 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.898+0000 7f9977fff700 1 --2- 172.21.15.45:0/3555641755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f996800b5f0 tx=0x7f9968006ee0).ready entity=mon.0 client_cookie=13cf2e3e3e8daf8a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.898+0000 7f9976ffd700 1 -- 172.21.15.45:0/3555641755 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f996800d600 con 0x7f9978070540 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.898+0000 7f9976ffd700 1 -- 172.21.15.45:0/3555641755 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f996800dc30 con 0x7f9978070540 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.898+0000 7f9976ffd700 1 -- 172.21.15.45:0/3555641755 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9968011d30 con 0x7f9978070540 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- 172.21.15.45:0/3555641755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 msgr2=0x7f99780fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 --2- 172.21.15.45:0/3555641755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f996800b5f0 tx=0x7f9968006ee0).stop 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- 172.21.15.45:0/3555641755 shutdown_connections 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 --2- 172.21.15.45:0/3555641755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fe9c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:57.903 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- 172.21.15.45:0/3555641755 >> 172.21.15.45:0/3555641755 conn(0x7f99780f6460 msgr2=0x7f99780f8880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- 172.21.15.45:0/3555641755 shutdown_connections 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- 172.21.15.45:0/3555641755 wait complete. 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 Processor -- start 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.899+0000 7f997e96f700 1 -- start start 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f997e96f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f997e96f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99780fa8c0 con 0x7f9978070540 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9977fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:57.904 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9977fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36892/0 (socket says 172.21.15.45:36892) 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9977fff700 1 -- 172.21.15.45:0/3452404542 learned_addr learned my addr 172.21.15.45:0/3452404542 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9977fff700 1 -- 172.21.15.45:0/3452404542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996800b200 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9977fff700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9968006b80 tx=0x7f9968006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f996800d9e0 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99780faa60 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f996801b8e0 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f996801a8c0 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.900+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99780faef0 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.901+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f996800f070 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.901+0000 7f9974ff9700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 0x7f99640323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.901+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f99680415a0 con 0x7f9978070540 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.902+0000 7f99777fe700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 0x7f99640323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:57.905 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.902+0000 7f99777fe700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 0x7f99640323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f996c0096a0 tx=0x7f996c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:57.906 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.903+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9978059100 con 0x7f9978070540 2024-02-19T22:13:57.910 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:57.908+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9968016070 con 0x7f9978070540 2024-02-19T22:13:58.120 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.117+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f9978045ad0 con 0x7f9978070540 2024-02-19T22:13:58.526 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.524+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f9968025980 con 0x7f9978070540 2024-02-19T22:13:58.528 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.525+0000 7f9974ff9700 1 -- 172.21.15.45:0/3452404542 <== mon.0 v2:172.21.15.45:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f9968010540 con 0x7f9978070540 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 msgr2=0x7f99640323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 0x7f99640323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f996c0096a0 tx=0x7f996c006b40).stop 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 msgr2=0x7f99780fc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9968006b80 tx=0x7f9968006ee0).stop 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 shutdown_connections 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7f996402ff20 0x7f99640323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 --2- 172.21.15.45:0/3452404542 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9978070540 0x7f99780fc440 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 >> 172.21.15.45:0/3452404542 conn(0x7f99780f6460 msgr2=0x7f99780f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 shutdown_connections 2024-02-19T22:13:58.531 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:58.527+0000 7f997e96f700 1 -- 172.21.15.45:0/3452404542 wait complete. 2024-02-19T22:13:58.559 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:57.379677+0000 mon.smithi045 (mon.0) 85 : audit [INF] from='mgr.14142 172.21.15.45:0/3554956054' entity='mgr.smithi045.bvlamo' 2024-02-19T22:13:58.560 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:58.119997+0000 mon.smithi045 (mon.0) 86 : audit [INF] from='client.? 172.21.15.45:0/3452404542' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T22:13:59.283 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:13:59.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:13:58.527474+0000 mon.smithi045 (mon.0) 87 : audit [INF] from='client.? 172.21.15.45:0/3452404542' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T22:13:59.543 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:13:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:13:58.527521+0000 mon.smithi045 (mon.0) 88 : cluster [DBG] mgrmap e13: smithi045.bvlamo(active, since 11s) 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.792+0000 7fbcd5b60700 1 Processor -- start 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.792+0000 7fbcd5b60700 1 -- start start 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbcd5b60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbcd5b60700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00fb6a0 con 0x7fbcd00fad90 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbccf7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbccf7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36934/0 (socket says 172.21.15.45:36934) 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbccf7fe700 1 -- 172.21.15.45:0/4116705832 learned_addr learned my addr 172.21.15.45:0/4116705832 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:59.824 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbccf7fe700 1 -- 172.21.15.45:0/4116705832 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd00fb7e0 con 0x7fbcd00fad90 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbccf7fe700 1 --2- 172.21.15.45:0/4116705832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbcc000b5f0 tx=0x7fbcc0006ee0).ready entity=mon.0 client_cookie=bf64edc878dd26f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbcce7fc700 1 -- 172.21.15.45:0/4116705832 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc000d600 con 0x7fbcd00fad90 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbcce7fc700 1 -- 172.21.15.45:0/4116705832 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcc000dc30 con 0x7fbcd00fad90 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.793+0000 7fbcce7fc700 1 -- 172.21.15.45:0/4116705832 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc0011d30 con 0x7fbcd00fad90 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 -- 172.21.15.45:0/4116705832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 msgr2=0x7fbcd00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/4116705832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fbcc000b5f0 tx=0x7fbcc0006ee0).stop 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 -- 172.21.15.45:0/4116705832 shutdown_connections 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/4116705832 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd00fb1b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 -- 172.21.15.45:0/4116705832 >> 172.21.15.45:0/4116705832 conn(0x7fbcd00f6530 msgr2=0x7fbcd00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 -- 172.21.15.45:0/4116705832 shutdown_connections 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 -- 172.21.15.45:0/4116705832 wait complete. 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.794+0000 7fbcd5b60700 1 Processor -- start 2024-02-19T22:13:59.825 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbcd5b60700 1 -- start start 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbcd5b60700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbcd5b60700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd00ff530 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbccf7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbccf7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36948/0 (socket says 172.21.15.45:36948) 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbccf7fe700 1 -- 172.21.15.45:0/604076581 learned_addr learned my addr 172.21.15.45:0/604076581 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbccf7fe700 1 -- 172.21.15.45:0/604076581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc000b200 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbccf7fe700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbcc0006b80 tx=0x7fbcc0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.795+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc000d9e0 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd00ff6d0 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbcc001b8e0 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbcc001a870 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd00ffb60 con 0x7fbcd00fad90 2024-02-19T22:13:59.826 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbcc000f070 con 0x7fbcd00fad90 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcb7fff700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 0x7fbcbc032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.796+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd0045ad0 con 0x7fbcd00fad90 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.797+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbcc0042450 con 0x7fbcd00fad90 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.797+0000 7fbcceffd700 1 -- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 msgr2=0x7fbcbc032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.797+0000 7fbcceffd700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 0x7fbcbc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:59.827 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.802+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbcc0016070 con 0x7fbcd00fad90 2024-02-19T22:14:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.998+0000 7fbcceffd700 1 -- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 msgr2=0x7fbcbc032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:14:00.000 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:13:59.998+0000 7fbcceffd700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 0x7fbcbc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:14:00.048 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.046+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fbcd00fcc60 con 0x7fbcd00fad90 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.047+0000 7fbcb7fff700 1 -- 172.21.15.45:0/604076581 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fbcc0022960 con 0x7fbcd00fad90 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "active_name": "smithi045.bvlamo", 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:14:00.049 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 msgr2=0x7fbcbc032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 0x7fbcbc032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 msgr2=0x7fbcd01010b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fbcc0006b80 tx=0x7fbcc0006ee0).stop 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 shutdown_connections 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.048+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fbcbc02ff70 0x7fbcbc032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.049+0000 7fbcd5b60700 1 --2- 172.21.15.45:0/604076581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbcd00fad90 0x7fbcd01010b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.049+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 >> 172.21.15.45:0/604076581 conn(0x7fbcd00f6530 msgr2=0x7fbcd0070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.049+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 shutdown_connections 2024-02-19T22:14:00.051 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:00.049+0000 7fbcd5b60700 1 -- 172.21.15.45:0/604076581 wait complete. 2024-02-19T22:14:00.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:00.047836+0000 mon.smithi045 (mon.0) 89 : audit [DBG] from='client.? 172.21.15.45:0/604076581' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:14:00.708 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for the mgr to restart... 2024-02-19T22:14:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:Waiting for mgr epoch 13... 2024-02-19T22:14:00.709 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:14:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce197d8700 1 Processor -- start 2024-02-19T22:14:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce197d8700 1 -- start start 2024-02-19T22:14:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce197d8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce197d8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce140fedc0 con 0x7fce140703f0 2024-02-19T22:14:01.339 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36952/0 (socket says 172.21.15.45:36952) 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.336+0000 7fce13fff700 1 -- 172.21.15.45:0/2899145837 learned_addr learned my addr 172.21.15.45:0/2899145837 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.337+0000 7fce13fff700 1 -- 172.21.15.45:0/2899145837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce140fef00 con 0x7fce140703f0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.337+0000 7fce13fff700 1 --2- 172.21.15.45:0/2899145837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fce0400b5f0 tx=0x7fce04006ee0).ready entity=mon.0 client_cookie=94ab508ca3bf9227 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.337+0000 7fce12ffd700 1 -- 172.21.15.45:0/2899145837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce0400d600 con 0x7fce140703f0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.337+0000 7fce12ffd700 1 -- 172.21.15.45:0/2899145837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce0400dc30 con 0x7fce140703f0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.337+0000 7fce12ffd700 1 -- 172.21.15.45:0/2899145837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04011d60 con 0x7fce140703f0 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 -- 172.21.15.45:0/2899145837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 msgr2=0x7fce140fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 --2- 172.21.15.45:0/2899145837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fce0400b5f0 tx=0x7fce04006ee0).stop 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 -- 172.21.15.45:0/2899145837 shutdown_connections 2024-02-19T22:14:01.340 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 --2- 172.21.15.45:0/2899145837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fe8d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:01.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 -- 172.21.15.45:0/2899145837 >> 172.21.15.45:0/2899145837 conn(0x7fce140f6390 msgr2=0x7fce140f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:01.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 -- 172.21.15.45:0/2899145837 shutdown_connections 2024-02-19T22:14:01.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 -- 172.21.15.45:0/2899145837 wait complete. 2024-02-19T22:14:01.342 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.338+0000 7fce197d8700 1 Processor -- start 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce197d8700 1 -- start start 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce197d8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce197d8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce140fa7d0 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36968/0 (socket says 172.21.15.45:36968) 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce13fff700 1 -- 172.21.15.45:0/2506304819 learned_addr learned my addr 172.21.15.45:0/2506304819 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.339+0000 7fce13fff700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0400b200 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.340+0000 7fce13fff700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fce0400e040 tx=0x7fce04006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.340+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce04011e10 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.340+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce0401b8e0 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.340+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce0401a930 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.341+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce140fa970 con 0x7fce140703f0 2024-02-19T22:14:01.343 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.341+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce140fae00 con 0x7fce140703f0 2024-02-19T22:14:01.344 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fce0400f070 con 0x7fce140703f0 2024-02-19T22:14:01.344 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce10ff9700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:01.344 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce04011610 con 0x7fce0002ff70 2024-02-19T22:14:01.345 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fce04040f70 con 0x7fce140703f0 2024-02-19T22:14:01.345 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce137fe700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 msgr2=0x7fce00032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:14:01.345 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.342+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:14:01.545 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.543+0000 7fce137fe700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 msgr2=0x7fce00032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:14:01.546 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.543+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:14:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.944+0000 7fce137fe700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 msgr2=0x7fce00032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:14:01.946 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:01.944+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:14:02.747 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:02.745+0000 7fce137fe700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 msgr2=0x7fce00032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.45:6800/172486393 2024-02-19T22:14:02.748 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:02.745+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:14:03.779 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:03.778+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fce0401ba50 con 0x7fce140703f0 2024-02-19T22:14:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:03.778+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 msgr2=0x7fce00032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:14:03.780 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:03.778+0000 7fce10ff9700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:14:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.779+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fce0401b5e0 con 0x7fce140703f0 2024-02-19T22:14:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.779+0000 7fce10ff9700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 0x7fce00035e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.779+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fce04011610 con 0x7fce00033a90 2024-02-19T22:14:04.781 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.779+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 0x7fce00035e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:04.782 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.780+0000 7fce137fe700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 0x7fce00035e80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fce080036d0 tx=0x7fce08006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:04.782 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.781+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fce04011610 con 0x7fce00033a90 2024-02-19T22:14:04.788 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fce14059100 con 0x7fce00033a90 2024-02-19T22:14:04.789 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce10ff9700 1 -- 172.21.15.45:0/2506304819 <== mgr.14162 v2:172.21.15.45:6800/3887107005 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fce14059100 con 0x7fce00033a90 2024-02-19T22:14:04.789 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: { 2024-02-19T22:14:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T22:14:04.790 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: } 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 msgr2=0x7fce00035e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 0x7fce00035e80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fce080036d0 tx=0x7fce08006ca0).stop 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 msgr2=0x7fce140fc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fce0400e040 tx=0x7fce04006ee0).stop 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 shutdown_connections 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fce00033a90 0x7fce00035e80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fce140703f0 0x7fce140fc350 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.787+0000 7fce197d8700 1 --2- 172.21.15.45:0/2506304819 >> [v2:172.21.15.45:6800/172486393,v1:172.21.15.45:6801/172486393] conn(0x7fce0002ff70 0x7fce00032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.788+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 >> 172.21.15.45:0/2506304819 conn(0x7fce140f6390 msgr2=0x7fce140f7de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.788+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 shutdown_connections 2024-02-19T22:14:04.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:04.788+0000 7fce197d8700 1 -- 172.21.15.45:0/2506304819 wait complete. 2024-02-19T22:14:05.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:03.726930+0000 mon.smithi045 (mon.0) 90 : cluster [INF] Active manager daemon smithi045.bvlamo restarted 2024-02-19T22:14:05.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:03.727312+0000 mon.smithi045 (mon.0) 91 : cluster [INF] Activating manager daemon smithi045.bvlamo 2024-02-19T22:14:05.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:03.728987+0000 mon.smithi045 (mon.0) 92 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:03.779797+0000 mon.smithi045 (mon.0) 93 : cluster [DBG] mgrmap e14: smithi045.bvlamo(active, starting, since 0.052568s) 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.780740+0000 mon.smithi045 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata", "id": "smithi045"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.781321+0000 mon.smithi045 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mgr metadata", "who": "smithi045.bvlamo", "id": "smithi045.bvlamo"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.782197+0000 mon.smithi045 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.782384+0000 mon.smithi045 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.782555+0000 mon.smithi045 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:03.796370+0000 mon.smithi045 (mon.0) 99 : cluster [INF] Manager daemon smithi045.bvlamo is now available 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.899682+0000 mon.smithi045 (mon.0) 100 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.901450+0000 mon.smithi045 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.927386+0000 mon.smithi045 (mon.0) 102 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:14:05.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:03.930198+0000 mon.smithi045 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:14:05.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:04.781632+0000 mgr.smithi045.bvlamo (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:14:05.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:04.782114+0000 mon.smithi045 (mon.0) 104 : cluster [DBG] mgrmap e15: smithi045.bvlamo(active, since 1.05488s) 2024-02-19T22:14:05.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:04.788325+0000 mgr.smithi045.bvlamo (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:14:05.824 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:05.169005+0000 mon.smithi045 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:06.128 INFO:teuthology.orchestra.run.smithi045.stderr:mgr epoch 13 is available 2024-02-19T22:14:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:Generating a dashboard self-signed certificate... 2024-02-19T22:14:06.129 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T22:14:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.172+0000 7f3585149700 1 Processor -- start 2024-02-19T22:14:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.172+0000 7f3585149700 1 -- start start 2024-02-19T22:14:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.172+0000 7f3585149700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:07.178 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.172+0000 7f3585149700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800fed80 con 0x7f3580070430 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.172+0000 7f357ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43396/0 (socket says 172.21.15.45:43396) 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357ffff700 1 -- 172.21.15.45:0/2079111840 learned_addr learned my addr 172.21.15.45:0/2079111840 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357ffff700 1 -- 172.21.15.45:0/2079111840 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35800feec0 con 0x7f3580070430 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357ffff700 1 --2- 172.21.15.45:0/2079111840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f357000b5f0 tx=0x7f3570006ee0).ready entity=mon.0 client_cookie=7aedac5627e5dad8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357effd700 1 -- 172.21.15.45:0/2079111840 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357000d600 con 0x7f3580070430 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357effd700 1 -- 172.21.15.45:0/2079111840 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f357000dc30 con 0x7f3580070430 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.173+0000 7f357effd700 1 -- 172.21.15.45:0/2079111840 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3570011d30 con 0x7f3580070430 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 -- 172.21.15.45:0/2079111840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 msgr2=0x7f35800fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 --2- 172.21.15.45:0/2079111840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f357000b5f0 tx=0x7f3570006ee0).stop 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 -- 172.21.15.45:0/2079111840 shutdown_connections 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 --2- 172.21.15.45:0/2079111840 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fe890 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 -- 172.21.15.45:0/2079111840 >> 172.21.15.45:0/2079111840 conn(0x7f35800f6390 msgr2=0x7f35800f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:07.179 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 -- 172.21.15.45:0/2079111840 shutdown_connections 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 -- 172.21.15.45:0/2079111840 wait complete. 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.174+0000 7f3585149700 1 Processor -- start 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f3585149700 1 -- start start 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f3585149700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f3585149700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35800fa790 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43406/0 (socket says 172.21.15.45:43406) 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357ffff700 1 -- 172.21.15.45:0/2548322535 learned_addr learned my addr 172.21.15.45:0/2548322535 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357ffff700 1 -- 172.21.15.45:0/2548322535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357000b200 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357ffff700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f357000e040 tx=0x7f3570006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.175+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357000da60 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.176+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35800fa930 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.176+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f357001b8e0 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.176+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f357001a900 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.176+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35800fadc0 con 0x7f3580070430 2024-02-19T22:14:07.180 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f357000f070 con 0x7f3580070430 2024-02-19T22:14:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f357cff9700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 0x7f356c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35700414e0 con 0x7f3580070430 2024-02-19T22:14:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f357f7fe700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 0x7f356c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3580059100 con 0x7f3580070430 2024-02-19T22:14:07.181 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.177+0000 7f357f7fe700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 0x7f356c032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f35740096a0 tx=0x7f3574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:07.183 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.182+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3570022310 con 0x7f3580070430 2024-02-19T22:14:07.352 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.350+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f35800ffce0 con 0x7f356c02ff70 2024-02-19T22:14:07.433 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.431+0000 7f357cff9700 1 -- 172.21.15.45:0/2548322535 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f35800ffce0 con 0x7f356c02ff70 2024-02-19T22:14:07.433 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T22:14:07.435 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 msgr2=0x7f356c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:07.435 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 0x7f356c032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f35740096a0 tx=0x7f3574006b40).stop 2024-02-19T22:14:07.435 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 msgr2=0x7f35800fc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f357000e040 tx=0x7f3570006ee0).stop 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 shutdown_connections 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f356c02ff70 0x7f356c032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 --2- 172.21.15.45:0/2548322535 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3580070430 0x7f35800fc310 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 >> 172.21.15.45:0/2548322535 conn(0x7f35800f6390 msgr2=0x7f35800f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 shutdown_connections 2024-02-19T22:14:07.436 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:07.433+0000 7f3585149700 1 -- 172.21.15.45:0/2548322535 wait complete. 2024-02-19T22:14:07.454 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:06.170397+0000 mon.smithi045 (mon.0) 106 : cluster [DBG] mgrmap e16: smithi045.bvlamo(active, since 2s) 2024-02-19T22:14:08.246 INFO:teuthology.orchestra.run.smithi045.stderr:Creating initial admin user... 2024-02-19T22:14:08.247 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpemh3gtuy:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T22:14:08.530 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:07.352048+0000 mgr.smithi045.bvlamo (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:14:08.530 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:07.430750+0000 mon.smithi045 (mon.0) 107 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:08.530 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:07.432430+0000 mon.smithi045 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:08.834 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.827+0000 7fefb6bf2700 1 Processor -- start 2024-02-19T22:14:08.834 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.827+0000 7fefb6bf2700 1 -- start start 2024-02-19T22:14:08.834 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.827+0000 7fefb6bf2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:08.834 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.827+0000 7fefb6bf2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb00fc840 con 0x7fefb0070430 2024-02-19T22:14:08.834 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefb498e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefb498e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43420/0 (socket says 172.21.15.45:43420) 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefb498e700 1 -- 172.21.15.45:0/2896130733 learned_addr learned my addr 172.21.15.45:0/2896130733 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefb498e700 1 -- 172.21.15.45:0/2896130733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0101340 con 0x7fefb0070430 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefb498e700 1 --2- 172.21.15.45:0/2896130733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fefa400b5f0 tx=0x7fefa4006ee0).ready entity=mon.0 client_cookie=f4820d1dba51f19d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefaf7fe700 1 -- 172.21.15.45:0/2896130733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa400d600 con 0x7fefb0070430 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefaf7fe700 1 -- 172.21.15.45:0/2896130733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefa400dc30 con 0x7fefb0070430 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.828+0000 7fefaf7fe700 1 -- 172.21.15.45:0/2896130733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa4011d30 con 0x7fefb0070430 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- 172.21.15.45:0/2896130733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 msgr2=0x7fefb0100e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/2896130733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fefa400b5f0 tx=0x7fefa4006ee0).stop 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- 172.21.15.45:0/2896130733 shutdown_connections 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/2896130733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb0100e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- 172.21.15.45:0/2896130733 >> 172.21.15.45:0/2896130733 conn(0x7fefb00f6530 msgr2=0x7fefb00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- 172.21.15.45:0/2896130733 shutdown_connections 2024-02-19T22:14:08.835 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- 172.21.15.45:0/2896130733 wait complete. 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 Processor -- start 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.829+0000 7fefb6bf2700 1 -- start start 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb6bf2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb6bf2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb010d330 con 0x7fefb0070430 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb498e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb498e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43422/0 (socket says 172.21.15.45:43422) 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb498e700 1 -- 172.21.15.45:0/774354224 learned_addr learned my addr 172.21.15.45:0/774354224 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb498e700 1 -- 172.21.15.45:0/774354224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa400b200 con 0x7fefb0070430 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb498e700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fefa400e040 tx=0x7fefa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa400d9e0 con 0x7fefb0070430 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefa401b8e0 con 0x7fefb0070430 2024-02-19T22:14:08.836 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb010d530 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.830+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa401a820 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.831+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb010e140 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.831+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fefa400f070 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.831+0000 7fefad7fa700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 0x7fef94032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.831+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefa40415e0 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.832+0000 7fefaffff700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 0x7fef94032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.832+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb0059100 con 0x7fefb0070430 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.832+0000 7fefaffff700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 0x7fef94032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:08.837 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:08.836+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefa4016070 con 0x7fefb0070430 2024-02-19T22:14:09.017 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.015+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fefb0058f20 con 0x7fef9402ffc0 2024-02-19T22:14:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.264+0000 7fefad7fa700 1 -- 172.21.15.45:0/774354224 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fefb0058f20 con 0x7fef9402ffc0 2024-02-19T22:14:09.266 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$lhAmfR/cp4FQiG8xpc9wU.kXS8C1wtKSCNHWR0zF1pY1vo8FA.n1y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708380849, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T22:14:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 msgr2=0x7fef94032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 0x7fef94032480 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).stop 2024-02-19T22:14:09.268 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 msgr2=0x7fefb010ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fefa400e040 tx=0x7fefa4006ee0).stop 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 shutdown_connections 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fef9402ffc0 0x7fef94032480 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 --2- 172.21.15.45:0/774354224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fefb0070430 0x7fefb010ce40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 >> 172.21.15.45:0/774354224 conn(0x7fefb00f6530 msgr2=0x7fefb00f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 shutdown_connections 2024-02-19T22:14:09.269 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:09.266+0000 7fefb6bf2700 1 -- 172.21.15.45:0/774354224 wait complete. 2024-02-19T22:14:10.046 INFO:teuthology.orchestra.run.smithi045.stderr:Fetching dashboard port number... 2024-02-19T22:14:10.046 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmp0319umfa:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph7b0n6i4:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T22:14:10.311 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:09.016893+0000 mgr.smithi045.bvlamo (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:14:10.311 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:09.265350+0000 mon.smithi045 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29bc488700 1 Processor -- start 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29bc488700 1 -- start start 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29bc488700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29bc488700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b40fc8a0 con 0x7f29b40703d0 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29ba224700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29ba224700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43428/0 (socket says 172.21.15.45:43428) 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.620+0000 7f29ba224700 1 -- 172.21.15.45:0/823702941 learned_addr learned my addr 172.21.15.45:0/823702941 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29ba224700 1 -- 172.21.15.45:0/823702941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b41013a0 con 0x7f29b40703d0 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29ba224700 1 --2- 172.21.15.45:0/823702941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f29b000b5f0 tx=0x7f29b0006ee0).ready entity=mon.0 client_cookie=490b8059800237e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29b9222700 1 -- 172.21.15.45:0/823702941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b000d600 con 0x7f29b40703d0 2024-02-19T22:14:10.633 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29b9222700 1 -- 172.21.15.45:0/823702941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b000dc30 con 0x7f29b40703d0 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29b9222700 1 -- 172.21.15.45:0/823702941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b0011d30 con 0x7f29b40703d0 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 -- 172.21.15.45:0/823702941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 msgr2=0x7f29b4100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 --2- 172.21.15.45:0/823702941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f29b000b5f0 tx=0x7f29b0006ee0).stop 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 -- 172.21.15.45:0/823702941 shutdown_connections 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 --2- 172.21.15.45:0/823702941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b4100eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 -- 172.21.15.45:0/823702941 >> 172.21.15.45:0/823702941 conn(0x7f29b40f6530 msgr2=0x7f29b40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.621+0000 7f29bc488700 1 -- 172.21.15.45:0/823702941 shutdown_connections 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.622+0000 7f29bc488700 1 -- 172.21.15.45:0/823702941 wait complete. 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.622+0000 7f29bc488700 1 Processor -- start 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.622+0000 7f29bc488700 1 -- start start 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.622+0000 7f29bc488700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29bc488700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b410d3a0 con 0x7f29b40703d0 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29ba224700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.634 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29ba224700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43430/0 (socket says 172.21.15.45:43430) 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29ba224700 1 -- 172.21.15.45:0/1323044925 learned_addr learned my addr 172.21.15.45:0/1323044925 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29ba224700 1 -- 172.21.15.45:0/1323044925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b000b200 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29ba224700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f29b000e040 tx=0x7f29b0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b000d9e0 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b410d5a0 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29b001b8e0 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29b001a820 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.623+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b410e1b0 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.624+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29b000f070 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.624+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b4059100 con 0x7f29b40703d0 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.624+0000 7f29a6ffd700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 0x7f299c0324e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:10.635 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.624+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29b0041650 con 0x7f29b40703d0 2024-02-19T22:14:10.636 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.625+0000 7f29b9a23700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 0x7f299c0324e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:10.636 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.625+0000 7f29b9a23700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 0x7f299c0324e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:10.636 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.629+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29b0019c90 con 0x7f29b40703d0 2024-02-19T22:14:10.788 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.787+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f29b410efe0 con 0x7f29b40703d0 2024-02-19T22:14:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.787+0000 7f29a6ffd700 1 -- 172.21.15.45:0/1323044925 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f29b0016070 con 0x7f29b40703d0 2024-02-19T22:14:10.789 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 8443 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 msgr2=0x7f299c0324e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 0x7f299c0324e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29a80096a0 tx=0x7f29a8006b40).stop 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 msgr2=0x7f29b410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f29b000e040 tx=0x7f29b0006ee0).stop 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 shutdown_connections 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f299c030020 0x7f299c0324e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.791 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 --2- 172.21.15.45:0/1323044925 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29b40703d0 0x7f29b410ceb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 >> 172.21.15.45:0/1323044925 conn(0x7f29b40f6530 msgr2=0x7f29b40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 shutdown_connections 2024-02-19T22:14:10.792 INFO:teuthology.orchestra.run.smithi045.stderr:/usr/bin/ceph: 2024-02-19T22:14:10.789+0000 7f29bc488700 1 -- 172.21.15.45:0/1323044925 wait complete. 2024-02-19T22:14:11.510 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:14:11.512 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:10.267278+0000 mon.smithi045 (mon.0) 110 : cluster [DBG] mgrmap e17: smithi045.bvlamo(active, since 6s) 2024-02-19T22:14:11.512 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:10.788300+0000 mon.smithi045 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.45:0/1323044925' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T22:14:11.519 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: disabled 2024-02-19T22:14:11.520 INFO:teuthology.orchestra.run.smithi045.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:14:11.527 INFO:teuthology.orchestra.run.smithi045.stderr:systemctl: inactive 2024-02-19T22:14:11.528 INFO:teuthology.orchestra.run.smithi045.stderr:firewalld.service is not enabled 2024-02-19T22:14:11.528 INFO:teuthology.orchestra.run.smithi045.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr:Ceph Dashboard is now available at: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: URL: https://smithi045.front.sepia.ceph.com:8443/ 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: User: admin 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: Password: ev14wf3hkx 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr:You can access the Ceph CLI with: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: ceph telemetry on 2024-02-19T22:14:11.532 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr:For more information see: 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr:Bootstrap complete. 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr:Releasing lock 139829399501288 on /run/cephadm/f6020aec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:14:11.533 INFO:teuthology.orchestra.run.smithi045.stderr:Lock 139829399501288 released on /run/cephadm/f6020aec-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:14:11.569 INFO:tasks.cephadm:Fetching config... 2024-02-19T22:14:11.569 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:14:11.569 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T22:14:11.589 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T22:14:11.589 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:14:11.590 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T22:14:11.647 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T22:14:11.647 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:14:11.648 DEBUG:teuthology.orchestra.run.smithi045:> sudo dd if=/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/mon.smithi045/keyring of=/dev/stdout 2024-02-19T22:14:11.715 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T22:14:11.715 DEBUG:teuthology.orchestra.run.smithi045:> set -ex 2024-02-19T22:14:11.715 DEBUG:teuthology.orchestra.run.smithi045:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T22:14:11.775 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T22:14:11.775 DEBUG:teuthology.orchestra.run.smithi045:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqUbJ4tb5VYHd7111Rm7h4gbbhYiYlorB4Nz8uikTBRx6REcJs1gW/PiWsNhnbo2QNNRawM7JJS0Wz/srPDjZdFbCWAycoepA13P+GQ8K0Q92pxwhgkr80ln+jbTLAUNDpa0ymoBs9tQaUor49u5bMyZVvDIR4EOKvgY2tUdp0L+kxjL9YyOjNAxWgq0el7lQl3oo6P55C1QcZVI6lAX8X+HoVw50r53a3J5JDEf9vAc3f3H50HEkwSje1F/sPfaoWs1MTI1RRWxvVizXZ3bCjK/R3VPXdwX0NR4DlxS9h7gDwbqBw8MP7Fbq+0WNzf2tE9TKtrV3umt7CLJ/pEftDvqNAoVUZT+GP00dwlNtIVLlOLDFlGLJ4/9k/fSSLkgowJ2cmLJ+S6PrJA5cikXyIzqSN58A8vxnmqz5/xIuTYSt2ZGn2yadUnXLKh5W2KffI2Cjesa7zFbJ/0u/D/xnGCBhbv6ljyFu3EQVxBYVDrzx7lSKjVaGDYSDgZN2mjeU= ceph-f6020aec-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:14:11.851 INFO:teuthology.orchestra.run.smithi045.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqUbJ4tb5VYHd7111Rm7h4gbbhYiYlorB4Nz8uikTBRx6REcJs1gW/PiWsNhnbo2QNNRawM7JJS0Wz/srPDjZdFbCWAycoepA13P+GQ8K0Q92pxwhgkr80ln+jbTLAUNDpa0ymoBs9tQaUor49u5bMyZVvDIR4EOKvgY2tUdp0L+kxjL9YyOjNAxWgq0el7lQl3oo6P55C1QcZVI6lAX8X+HoVw50r53a3J5JDEf9vAc3f3H50HEkwSje1F/sPfaoWs1MTI1RRWxvVizXZ3bCjK/R3VPXdwX0NR4DlxS9h7gDwbqBw8MP7Fbq+0WNzf2tE9TKtrV3umt7CLJ/pEftDvqNAoVUZT+GP00dwlNtIVLlOLDFlGLJ4/9k/fSSLkgowJ2cmLJ+S6PrJA5cikXyIzqSN58A8vxnmqz5/xIuTYSt2ZGn2yadUnXLKh5W2KffI2Cjesa7zFbJ/0u/D/xnGCBhbv6ljyFu3EQVxBYVDrzx7lSKjVaGDYSDgZN2mjeU= ceph-f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:14:11.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqUbJ4tb5VYHd7111Rm7h4gbbhYiYlorB4Nz8uikTBRx6REcJs1gW/PiWsNhnbo2QNNRawM7JJS0Wz/srPDjZdFbCWAycoepA13P+GQ8K0Q92pxwhgkr80ln+jbTLAUNDpa0ymoBs9tQaUor49u5bMyZVvDIR4EOKvgY2tUdp0L+kxjL9YyOjNAxWgq0el7lQl3oo6P55C1QcZVI6lAX8X+HoVw50r53a3J5JDEf9vAc3f3H50HEkwSje1F/sPfaoWs1MTI1RRWxvVizXZ3bCjK/R3VPXdwX0NR4DlxS9h7gDwbqBw8MP7Fbq+0WNzf2tE9TKtrV3umt7CLJ/pEftDvqNAoVUZT+GP00dwlNtIVLlOLDFlGLJ4/9k/fSSLkgowJ2cmLJ+S6PrJA5cikXyIzqSN58A8vxnmqz5/xIuTYSt2ZGn2yadUnXLKh5W2KffI2Cjesa7zFbJ/0u/D/xnGCBhbv6ljyFu3EQVxBYVDrzx7lSKjVaGDYSDgZN2mjeU= ceph-f6020aec-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:14:11.906 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqUbJ4tb5VYHd7111Rm7h4gbbhYiYlorB4Nz8uikTBRx6REcJs1gW/PiWsNhnbo2QNNRawM7JJS0Wz/srPDjZdFbCWAycoepA13P+GQ8K0Q92pxwhgkr80ln+jbTLAUNDpa0ymoBs9tQaUor49u5bMyZVvDIR4EOKvgY2tUdp0L+kxjL9YyOjNAxWgq0el7lQl3oo6P55C1QcZVI6lAX8X+HoVw50r53a3J5JDEf9vAc3f3H50HEkwSje1F/sPfaoWs1MTI1RRWxvVizXZ3bCjK/R3VPXdwX0NR4DlxS9h7gDwbqBw8MP7Fbq+0WNzf2tE9TKtrV3umt7CLJ/pEftDvqNAoVUZT+GP00dwlNtIVLlOLDFlGLJ4/9k/fSSLkgowJ2cmLJ+S6PrJA5cikXyIzqSN58A8vxnmqz5/xIuTYSt2ZGn2yadUnXLKh5W2KffI2Cjesa7zFbJ/0u/D/xnGCBhbv6ljyFu3EQVxBYVDrzx7lSKjVaGDYSDgZN2mjeU= ceph-f6020aec-cf73-11ee-95bb-87774f69a715 2024-02-19T22:14:11.918 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T22:14:14.565 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.548+0000 7fe7cd59c700 1 -- 172.21.15.45:0/2796566212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 msgr2=0x7fe7c80fb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.548+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/2796566212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c80fb3a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fe7b8009fb0 tx=0x7fe7b8009b30).stop 2024-02-19T22:14:14.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.548+0000 7fe7cd59c700 1 -- 172.21.15.45:0/2796566212 shutdown_connections 2024-02-19T22:14:14.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.548+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/2796566212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c80fb3a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.548+0000 7fe7cd59c700 1 -- 172.21.15.45:0/2796566212 >> 172.21.15.45:0/2796566212 conn(0x7fe7c80f09a0 msgr2=0x7fe7c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:14.566 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.549+0000 7fe7cd59c700 1 -- 172.21.15.45:0/2796566212 shutdown_connections 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.549+0000 7fe7cd59c700 1 -- 172.21.15.45:0/2796566212 wait complete. 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.549+0000 7fe7cd59c700 1 Processor -- start 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.549+0000 7fe7cd59c700 1 -- start start 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7cd59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7cd59c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8108cc0 con 0x7fe7c8060160 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:43456/0 (socket says 172.21.15.45:43456) 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c7fff700 1 -- 172.21.15.45:0/840069055 learned_addr learned my addr 172.21.15.45:0/840069055 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c7fff700 1 -- 172.21.15.45:0/840069055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b8005040 con 0x7fe7c8060160 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c7fff700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe7b8005010 tx=0x7fe7b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8009320 con 0x7fe7c8060160 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c8108ec0 con 0x7fe7c8060160 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe7b8013380 con 0x7fe7c8060160 2024-02-19T22:14:14.567 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.550+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7b8014e50 con 0x7fe7c8060160 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.551+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c8109350 con 0x7fe7c8060160 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.551+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe7b80263a0 con 0x7fe7c8060160 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.551+0000 7fe7c4ff9700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 0x7fe7b00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.551+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c8045ad0 con 0x7fe7c8060160 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.551+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe7b8014470 con 0x7fe7c8060160 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.552+0000 7fe7c77fe700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 0x7fe7b00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.552+0000 7fe7c77fe700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 0x7fe7b00323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe7bc0096a0 tx=0x7fe7bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:14.568 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.557+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7b8012920 con 0x7fe7c8060160 2024-02-19T22:14:14.726 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.724+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fe7c80fe020 con 0x7fe7c8060160 2024-02-19T22:14:14.728 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.727+0000 7fe7c4ff9700 1 -- 172.21.15.45:0/840069055 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fe7b802b020 con 0x7fe7c8060160 2024-02-19T22:14:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 msgr2=0x7fe7b00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 0x7fe7b00323d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fe7bc0096a0 tx=0x7fe7bc006b40).stop 2024-02-19T22:14:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 msgr2=0x7fe7c81087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fe7b8005010 tx=0x7fe7b800b040).stop 2024-02-19T22:14:14.730 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 shutdown_connections 2024-02-19T22:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe7b002ff10 0x7fe7b00323d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 --2- 172.21.15.45:0/840069055 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe7c8060160 0x7fe7c81087d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 >> 172.21.15.45:0/840069055 conn(0x7fe7c80f09a0 msgr2=0x7fe7c80f14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 shutdown_connections 2024-02-19T22:14:14.731 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:14.729+0000 7fe7cd59c700 1 -- 172.21.15.45:0/840069055 wait complete. 2024-02-19T22:14:15.349 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:14.071875+0000 mon.smithi045 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:15.350 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:14.384964+0000 mon.smithi045 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:15.350 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:14:14.391358+0000 mgr.smithi045.bvlamo (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi045 on smithi045 2024-02-19T22:14:15.350 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:14.728268+0000 mon.smithi045 (mon.0) 114 : audit [INF] from='client.? 172.21.15.45:0/840069055' entity='client.admin' 2024-02-19T22:14:15.370 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi097 2024-02-19T22:14:15.370 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:14:15.370 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.conf 2024-02-19T22:14:15.392 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-02-19T22:14:15.392 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T22:14:15.453 INFO:tasks.cephadm:Adding host smithi097 to orchestrator... 2024-02-19T22:14:15.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph orch host add smithi097 2024-02-19T22:14:25.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:23.783484+0000 mgr.smithi045.bvlamo (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:27.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:25.783805+0000 mgr.smithi045.bvlamo (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:27.784172+0000 mgr.smithi045.bvlamo (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:31.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:29.784510+0000 mgr.smithi045.bvlamo (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:33.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:31.784840+0000 mgr.smithi045.bvlamo (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:35.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:33.785084+0000 mgr.smithi045.bvlamo (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- 172.21.15.97:0/1339130428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 msgr2=0x7fa3fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 --2- 172.21.15.97:0/1339130428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc0f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa3f0009fb0 tx=0x7fa3f0009b30).stop 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- 172.21.15.97:0/1339130428 shutdown_connections 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 --2- 172.21.15.97:0/1339130428 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc0f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- 172.21.15.97:0/1339130428 >> 172.21.15.97:0/1339130428 conn(0x7fa3fc0f0b70 msgr2=0x7fa3fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- 172.21.15.97:0/1339130428 shutdown_connections 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- 172.21.15.97:0/1339130428 wait complete. 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 Processor -- start 2024-02-19T22:14:37.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- start start 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.741+0000 7fa4047e3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc060240 con 0x7fa3fc0f5410 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa40257f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa40257f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44550/0 (socket says 172.21.15.97:44550) 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa40257f700 1 -- 172.21.15.97:0/2048827238 learned_addr learned my addr 172.21.15.97:0/2048827238 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa40257f700 1 -- 172.21.15.97:0/2048827238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0005040 con 0x7fa3fc0f5410 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa40257f700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3f00089f0 tx=0x7fa3f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f0009320 con 0x7fa3fc0f5410 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc060440 con 0x7fa3fc0f5410 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3f0013380 con 0x7fa3fc0f5410 2024-02-19T22:14:37.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3f001ed80 con 0x7fa3fc0f5410 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.742+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc109b90 con 0x7fa3fc0f5410 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.743+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3f001e420 con 0x7fa3fc0f5410 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.743+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc045ad0 con 0x7fa3fc0f5410 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.743+0000 7fa3eeffd700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 0x7fa3e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.743+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3f0041190 con 0x7fa3fc0f5410 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.743+0000 7fa401d7e700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 0x7fa3e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:37.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.744+0000 7fa401d7e700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 0x7fa3e8032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3f80096a0 tx=0x7fa3f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:37.749 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.748+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3f001cb10 con 0x7fa3fc0f5410 2024-02-19T22:14:37.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:35.785400+0000 mgr.smithi045.bvlamo (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:37.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:37.917+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3fc0f1d20 con 0x7fa3e802ffb0 2024-02-19T22:14:38.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.521+0000 7fa3eeffd700 1 -- 172.21.15.97:0/2048827238 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fa3fc0f1d20 con 0x7fa3e802ffb0 2024-02-19T22:14:38.523 INFO:teuthology.orchestra.run.smithi097.stdout:Added host 'smithi097' 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 msgr2=0x7fa3e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 0x7fa3e8032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fa3f80096a0 tx=0x7fa3f8006b40).stop 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 msgr2=0x7fa3fc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fa3f00089f0 tx=0x7fa3f000b040).stop 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 shutdown_connections 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3e802ffb0 0x7fa3e8032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 --2- 172.21.15.97:0/2048827238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3fc0f5410 0x7fa3fc1094e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 >> 172.21.15.97:0/2048827238 conn(0x7fa3fc0f0b70 msgr2=0x7fa3fc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 shutdown_connections 2024-02-19T22:14:38.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:38.523+0000 7fa4047e3700 1 -- 172.21.15.97:0/2048827238 wait complete. 2024-02-19T22:14:39.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T22:14:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:37.785774+0000 mgr.smithi045.bvlamo (mgr.14162) 13 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:37.918862+0000 mgr.smithi045.bvlamo (mgr.14162) 14 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:14:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:38.521247+0000 mon.smithi045 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:39.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:14:38.521690+0000 mgr.smithi045.bvlamo (mgr.14162) 15 : cephadm [INF] Added host smithi097 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 -- 172.21.15.97:0/994433403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 msgr2=0x7fe3440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 --2- 172.21.15.97:0/994433403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe3440f90f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fe32c009fb0 tx=0x7fe32c009b30).stop 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 -- 172.21.15.97:0/994433403 shutdown_connections 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 --2- 172.21.15.97:0/994433403 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe3440f90f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 -- 172.21.15.97:0/994433403 >> 172.21.15.97:0/994433403 conn(0x7fe3440f0b70 msgr2=0x7fe3440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 -- 172.21.15.97:0/994433403 shutdown_connections 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.050+0000 7fe34a36d700 1 -- 172.21.15.97:0/994433403 wait complete. 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.051+0000 7fe34a36d700 1 Processor -- start 2024-02-19T22:14:41.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.051+0000 7fe34a36d700 1 -- start start 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.051+0000 7fe34a36d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.051+0000 7fe34a36d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe344106c70 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.051+0000 7fe343fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe343fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44568/0 (socket says 172.21.15.97:44568) 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe343fff700 1 -- 172.21.15.97:0/637467240 learned_addr learned my addr 172.21.15.97:0/637467240 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe343fff700 1 -- 172.21.15.97:0/637467240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c005040 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe343fff700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe32c0089f0 tx=0x7fe32c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe32c009320 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe344106e70 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe32c013380 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe32c014e20 con 0x7fe344060280 2024-02-19T22:14:41.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.052+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe344107300 con 0x7fe344060280 2024-02-19T22:14:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.053+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe32c014420 con 0x7fe344060280 2024-02-19T22:14:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.053+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe344045ad0 con 0x7fe344060280 2024-02-19T22:14:41.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.053+0000 7fe340ff9700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 0x7fe3300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.053+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe32c040fc0 con 0x7fe344060280 2024-02-19T22:14:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.053+0000 7fe3437fe700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 0x7fe3300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:41.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.054+0000 7fe3437fe700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 0x7fe3300323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe3340096a0 tx=0x7fe334006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:41.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.058+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe32c06d2b0 con 0x7fe344060280 2024-02-19T22:14:41.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.230+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3440f1cb0 con 0x7fe33002ff10 2024-02-19T22:14:41.233 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.232+0000 7fe340ff9700 1 -- 172.21.15.97:0/637467240 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+156 (secure 0 0 0) 0x7fe3440f1cb0 con 0x7fe33002ff10 2024-02-19T22:14:41.233 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:14:41.233 INFO:teuthology.orchestra.run.smithi097.stdout:[{"addr": "172.21.15.45", "hostname": "smithi045", "labels": [], "status": ""}, {"addr": "smithi097", "hostname": "smithi097", "labels": [], "status": ""}] 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 msgr2=0x7fe3300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 0x7fe3300323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fe3340096a0 tx=0x7fe334006b40).stop 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 msgr2=0x7fe344106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fe32c0089f0 tx=0x7fe32c00b040).stop 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 shutdown_connections 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe33002ff10 0x7fe3300323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 --2- 172.21.15.97:0/637467240 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe344060280 0x7fe344106780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 >> 172.21.15.97:0/637467240 conn(0x7fe3440f0b70 msgr2=0x7fe3440f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 shutdown_connections 2024-02-19T22:14:41.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:41.233+0000 7fe34a36d700 1 -- 172.21.15.97:0/637467240 wait complete. 2024-02-19T22:14:41.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:39.786139+0000 mgr.smithi045.bvlamo (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:41.904 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T22:14:41.904 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T22:14:42.640 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:41.231946+0000 mgr.smithi045.bvlamo (mgr.14162) 17 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T22:14:43.588 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:41.786506+0000 mgr.smithi045.bvlamo (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 -- 172.21.15.45:0/4214995249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 msgr2=0x7faf840fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 --2- 172.21.15.45:0/4214995249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf840fb380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7faf74009fb0 tx=0x7faf74009b30).stop 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 -- 172.21.15.45:0/4214995249 shutdown_connections 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 --2- 172.21.15.45:0/4214995249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf840fb380 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 -- 172.21.15.45:0/4214995249 >> 172.21.15.45:0/4214995249 conn(0x7faf840f09a0 msgr2=0x7faf840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 -- 172.21.15.45:0/4214995249 shutdown_connections 2024-02-19T22:14:43.808 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.806+0000 7faf88972700 1 -- 172.21.15.45:0/4214995249 wait complete. 2024-02-19T22:14:43.809 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf88972700 1 Processor -- start 2024-02-19T22:14:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf88972700 1 -- start start 2024-02-19T22:14:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf88972700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf88972700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf84108cc0 con 0x7faf84060180 2024-02-19T22:14:43.810 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.45:36682/0 (socket says 172.21.15.45:36682) 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.807+0000 7faf837fe700 1 -- 172.21.15.45:0/175007363 learned_addr learned my addr 172.21.15.45:0/175007363 (peer_addr_for_me v2:172.21.15.45:0/0) 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf837fe700 1 -- 172.21.15.45:0/175007363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74005040 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf837fe700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faf74005010 tx=0x7faf7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74009320 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf84108ec0 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf74013380 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf74014e50 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.808+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf84109350 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf740263a0 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf7bfff700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf74041290 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84045ad0 con 0x7faf84060180 2024-02-19T22:14:43.811 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf82ffd700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:43.812 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.809+0000 7faf82ffd700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 0x7faf700323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7faf6c0096a0 tx=0x7faf6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:43.814 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.813+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf74012920 con 0x7faf84060180 2024-02-19T22:14:43.980 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:43.978+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7faf840fe020 con 0x7faf84060180 2024-02-19T22:14:44.537 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.536+0000 7faf7bfff700 1 -- 172.21.15.45:0/175007363 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7faf7402c090 con 0x7faf84060180 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 msgr2=0x7faf700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 0x7faf700323d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7faf6c0096a0 tx=0x7faf6c006b40).stop 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 msgr2=0x7faf841087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7faf74005010 tx=0x7faf7400b040).stop 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 shutdown_connections 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf7002ff10 0x7faf700323d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 --2- 172.21.15.45:0/175007363 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf84060180 0x7faf841087d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 >> 172.21.15.45:0/175007363 conn(0x7faf840f09a0 msgr2=0x7faf840f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 shutdown_connections 2024-02-19T22:14:44.540 INFO:teuthology.orchestra.run.smithi045.stderr:2024-02-19T22:14:44.538+0000 7faf88972700 1 -- 172.21.15.45:0/175007363 wait complete. 2024-02-19T22:14:44.541 INFO:teuthology.orchestra.run.smithi045.stderr:adjusted tunables profile to default 2024-02-19T22:14:44.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:43.979647+0000 mon.smithi045 (mon.0) 116 : audit [INF] from='client.? 172.21.15.45:0/175007363' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T22:14:45.225 INFO:tasks.cephadm:Adding mon.smithi045 on smithi045 2024-02-19T22:14:45.225 INFO:tasks.cephadm:Adding mon.smithi097 on smithi097 2024-02-19T22:14:45.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi045:172.21.15.45=smithi045;smithi097:172.21.15.97=smithi097' 2024-02-19T22:14:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:43.786793+0000 mgr.smithi045.bvlamo (mgr.14162) 19 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:44.536956+0000 mon.smithi045 (mon.0) 117 : audit [INF] from='client.? 172.21.15.45:0/175007363' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T22:14:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:44.537013+0000 mon.smithi045 (mon.0) 118 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T22:14:47.075 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 -- 172.21.15.97:0/2740264953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 msgr2=0x7f6f040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 --2- 172.21.15.97:0/2740264953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f040f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f6ef8009fb0 tx=0x7f6ef8009b30).stop 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 -- 172.21.15.97:0/2740264953 shutdown_connections 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 --2- 172.21.15.97:0/2740264953 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f040f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 -- 172.21.15.97:0/2740264953 >> 172.21.15.97:0/2740264953 conn(0x7f6f040f0b70 msgr2=0x7f6f040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 -- 172.21.15.97:0/2740264953 shutdown_connections 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.073+0000 7f6f0b261700 1 -- 172.21.15.97:0/2740264953 wait complete. 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.074+0000 7f6f0b261700 1 Processor -- start 2024-02-19T22:14:47.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.074+0000 7f6f0b261700 1 -- start start 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f0b261700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f0b261700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f04105c00 con 0x7f6f040f5410 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f08ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f08ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58044/0 (socket says 172.21.15.97:58044) 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f08ffd700 1 -- 172.21.15.97:0/858271998 learned_addr learned my addr 172.21.15.97:0/858271998 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f08ffd700 1 -- 172.21.15.97:0/858271998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8005040 con 0x7f6f040f5410 2024-02-19T22:14:47.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f08ffd700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6ef8005010 tx=0x7f6ef800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ef8009320 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f04105e00 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ef8013380 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ef8014e50 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.075+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f04106290 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.076+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ef8014420 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.076+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f04045ad0 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.076+0000 7f6f017fa700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 0x7f6ef40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.076+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ef8040e60 con 0x7f6f040f5410 2024-02-19T22:14:47.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.077+0000 7f6f03fff700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 0x7f6ef40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:47.080 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.077+0000 7f6f03fff700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 0x7f6ef40323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6ef00096a0 tx=0x7f6ef0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:47.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.082+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ef801db70 con 0x7f6f040f5410 2024-02-19T22:14:47.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.256+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 --> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi045:172.21.15.45=smithi045;smithi097:172.21.15.97=smithi097", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f040f1da0 con 0x7f6ef402ff10 2024-02-19T22:14:47.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.262+0000 7f6f017fa700 1 -- 172.21.15.97:0/858271998 <== mgr.14162 v2:172.21.15.45:6800/3887107005 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6f040f1da0 con 0x7f6ef402ff10 2024-02-19T22:14:47.263 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mon update... 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 msgr2=0x7f6ef40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 0x7f6ef40323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6ef00096a0 tx=0x7f6ef0006b40).stop 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 msgr2=0x7f6f04105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6ef8005010 tx=0x7f6ef800b040).stop 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 shutdown_connections 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6ef402ff10 0x7f6ef40323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 --2- 172.21.15.97:0/858271998 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6f040f5410 0x7f6f04105710 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 >> 172.21.15.97:0/858271998 conn(0x7f6f040f0b70 msgr2=0x7f6f040f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:47.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 shutdown_connections 2024-02-19T22:14:47.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:47.263+0000 7f6f0b261700 1 -- 172.21.15.97:0/858271998 wait complete. 2024-02-19T22:14:47.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:45.787118+0000 mgr.smithi045.bvlamo (mgr.14162) 20 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:47.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:47.262293+0000 mon.smithi045 (mon.0) 119 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:14:47.955 DEBUG:teuthology.orchestra.run.smithi097:mon.smithi097> sudo journalctl -f -n 0 -u ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097.service 2024-02-19T22:14:47.957 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:47.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:47.986 INFO:journalctl@ceph.mon.smithi097.smithi097.stdout:-- Logs begin at Mon 2024-02-19 22:01:37 UTC. -- 2024-02-19T22:14:48.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:47.258267+0000 mgr.smithi045.bvlamo (mgr.14162) 21 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi045:172.21.15.45=smithi045;smithi097:172.21.15.97=smithi097", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:14:48.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:14:47.259772+0000 mgr.smithi045.bvlamo (mgr.14162) 22 : cephadm [INF] Saving service mon spec with placement smithi045:172.21.15.45=smithi045;smithi097:172.21.15.97=smithi097;count:2 2024-02-19T22:14:49.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 -- 172.21.15.97:0/2270237899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 msgr2=0x7fe30c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 --2- 172.21.15.97:0/2270237899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c0f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe300009fb0 tx=0x7fe300009b30).stop 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 -- 172.21.15.97:0/2270237899 shutdown_connections 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 --2- 172.21.15.97:0/2270237899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c0f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 -- 172.21.15.97:0/2270237899 >> 172.21.15.97:0/2270237899 conn(0x7fe30c0f0b70 msgr2=0x7fe30c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 -- 172.21.15.97:0/2270237899 shutdown_connections 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.728+0000 7fe3137fb700 1 -- 172.21.15.97:0/2270237899 wait complete. 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe3137fb700 1 Processor -- start 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe3137fb700 1 -- start start 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe3137fb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe3137fb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c105bd0 con 0x7fe30c0f5410 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe311597700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe311597700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58062/0 (socket says 172.21.15.97:58062) 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.729+0000 7fe311597700 1 -- 172.21.15.97:0/3911881472 learned_addr learned my addr 172.21.15.97:0/3911881472 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe311597700 1 -- 172.21.15.97:0/3911881472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe300005040 con 0x7fe30c0f5410 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe311597700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe300005010 tx=0x7fe30000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe300009320 con 0x7fe30c0f5410 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c105dd0 con 0x7fe30c0f5410 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe300013380 con 0x7fe30c0f5410 2024-02-19T22:14:49.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe300014e50 con 0x7fe30c0f5410 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.730+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c106260 con 0x7fe30c0f5410 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.731+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe300014420 con 0x7fe30c0f5410 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.731+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c045ad0 con 0x7fe30c0f5410 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.731+0000 7fe2fdffb700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 0x7fe2f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.731+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe300040ef0 con 0x7fe30c0f5410 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.732+0000 7fe310d96700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 0x7fe2f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:49.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.732+0000 7fe310d96700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 0x7fe2f80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe3080096a0 tx=0x7fe308006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:49.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.736+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe3000095f0 con 0x7fe30c0f5410 2024-02-19T22:14:49.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:47.787435+0000 mgr.smithi045.bvlamo (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:49.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.939+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe30c107110 con 0x7fe30c0f5410 2024-02-19T22:14:49.941 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.940+0000 7fe2fdffb700 1 -- 172.21.15.97:0/3911881472 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe30001db70 con 0x7fe30c0f5410 2024-02-19T22:14:49.941 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:14:49.941 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 msgr2=0x7fe2f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 0x7fe2f80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fe3080096a0 tx=0x7fe308006b40).stop 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 msgr2=0x7fe30c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe300005010 tx=0x7fe30000b040).stop 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 shutdown_connections 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe2f802ff10 0x7fe2f80323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 --2- 172.21.15.97:0/3911881472 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe30c0f5410 0x7fe30c1056e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 >> 172.21.15.97:0/3911881472 conn(0x7fe30c0f0b70 msgr2=0x7fe30c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.941+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 shutdown_connections 2024-02-19T22:14:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:49.942+0000 7fe3137fb700 1 -- 172.21.15.97:0/3911881472 wait complete. 2024-02-19T22:14:49.944 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:14:50.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:49.940687+0000 mon.smithi045 (mon.0) 120 : audit [DBG] from='client.? 172.21.15.97:0/3911881472' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:51.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:51.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:49.787784+0000 mgr.smithi045.bvlamo (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 -- 172.21.15.97:0/648081301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 msgr2=0x7fe5740f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 --2- 172.21.15.97:0/648081301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5740f4fa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fe568009fb0 tx=0x7fe568009b30).stop 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 -- 172.21.15.97:0/648081301 shutdown_connections 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 --2- 172.21.15.97:0/648081301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5740f4fa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 -- 172.21.15.97:0/648081301 >> 172.21.15.97:0/648081301 conn(0x7fe5740f02e0 msgr2=0x7fe5740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:53.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 -- 172.21.15.97:0/648081301 shutdown_connections 2024-02-19T22:14:53.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.524+0000 7fe579553700 1 -- 172.21.15.97:0/648081301 wait complete. 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe579553700 1 Processor -- start 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe579553700 1 -- start start 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe579553700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe579553700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe574105bb0 con 0x7fe5740f4b80 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe572ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58074/0 (socket says 172.21.15.97:58074) 2024-02-19T22:14:53.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.525+0000 7fe572ffd700 1 -- 172.21.15.97:0/334258153 learned_addr learned my addr 172.21.15.97:0/334258153 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe572ffd700 1 -- 172.21.15.97:0/334258153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568005040 con 0x7fe5740f4b80 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe572ffd700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe568005010 tx=0x7fe56800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe568009320 con 0x7fe5740f4b80 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe574105db0 con 0x7fe5740f4b80 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe568013380 con 0x7fe5740f4b80 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe568014e50 con 0x7fe5740f4b80 2024-02-19T22:14:53.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.526+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe574106240 con 0x7fe5740f4b80 2024-02-19T22:14:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.527+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5680095f0 con 0x7fe5740f4b80 2024-02-19T22:14:53.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.527+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe574045ad0 con 0x7fe5740f4b80 2024-02-19T22:14:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.527+0000 7fe55b7fe700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 0x7fe55c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.527+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe568040f10 con 0x7fe5740f4b80 2024-02-19T22:14:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.527+0000 7fe5727fc700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 0x7fe55c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:53.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.528+0000 7fe5727fc700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 0x7fe55c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe5640096a0 tx=0x7fe564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.532+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe568014420 con 0x7fe5740f4b80 2024-02-19T22:14:53.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.737+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5740fa710 con 0x7fe5740f4b80 2024-02-19T22:14:53.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.738+0000 7fe55b7fe700 1 -- 172.21.15.97:0/334258153 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe568010230 con 0x7fe5740f4b80 2024-02-19T22:14:53.739 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:14:53.739 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 msgr2=0x7fe55c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 0x7fe55c0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fe5640096a0 tx=0x7fe564006b40).stop 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 msgr2=0x7fe5741056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe568005010 tx=0x7fe56800b040).stop 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 shutdown_connections 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe55c02ff10 0x7fe55c0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 --2- 172.21.15.97:0/334258153 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe5740f4b80 0x7fe5741056c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 >> 172.21.15.97:0/334258153 conn(0x7fe5740f02e0 msgr2=0x7fe5740f24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 shutdown_connections 2024-02-19T22:14:53.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:53.739+0000 7fe579553700 1 -- 172.21.15.97:0/334258153 wait complete. 2024-02-19T22:14:53.742 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:14:53.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:51.788187+0000 mgr.smithi045.bvlamo (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:54.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:53.738643+0000 mon.smithi045 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.97:0/334258153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:55.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:55.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:55.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:53.788519+0000 mgr.smithi045.bvlamo (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 -- 172.21.15.97:0/856014228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 msgr2=0x7f93000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 --2- 172.21.15.97:0/856014228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93000f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f92f4009fb0 tx=0x7f92f4009b30).stop 2024-02-19T22:14:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 -- 172.21.15.97:0/856014228 shutdown_connections 2024-02-19T22:14:57.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 --2- 172.21.15.97:0/856014228 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93000f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 -- 172.21.15.97:0/856014228 >> 172.21.15.97:0/856014228 conn(0x7f93000f1c70 msgr2=0x7f93000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 -- 172.21.15.97:0/856014228 shutdown_connections 2024-02-19T22:14:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.529+0000 7f9306f63700 1 -- 172.21.15.97:0/856014228 wait complete. 2024-02-19T22:14:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.530+0000 7f9306f63700 1 Processor -- start 2024-02-19T22:14:57.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.530+0000 7f9306f63700 1 -- start start 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.530+0000 7f9306f63700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.530+0000 7f9306f63700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9300106cd0 con 0x7f93000f6510 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.530+0000 7f9304cff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9304cff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46666/0 (socket says 172.21.15.97:46666) 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9304cff700 1 -- 172.21.15.97:0/3712147646 learned_addr learned my addr 172.21.15.97:0/3712147646 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9304cff700 1 -- 172.21.15.97:0/3712147646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f4005040 con 0x7f93000f6510 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9304cff700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f92f4005010 tx=0x7f92f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92f4009320 con 0x7f93000f6510 2024-02-19T22:14:57.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9300106ed0 con 0x7f93000f6510 2024-02-19T22:14:57.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9300107360 con 0x7f93000f6510 2024-02-19T22:14:57.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92f4013380 con 0x7f93000f6510 2024-02-19T22:14:57.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.531+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92f4014e50 con 0x7f93000f6510 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.532+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92f4014420 con 0x7f93000f6510 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.532+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9300045ad0 con 0x7f93000f6510 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.532+0000 7f92fd7fa700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 0x7f92f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.532+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92f4040fb0 con 0x7f93000f6510 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.532+0000 7f92fffff700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 0x7f92f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.533+0000 7f92fffff700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 0x7f92f00323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f92ec0096a0 tx=0x7f92ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:57.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.537+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92f4014790 con 0x7f93000f6510 2024-02-19T22:14:57.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.740+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93001081b0 con 0x7f93000f6510 2024-02-19T22:14:57.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.741+0000 7f92fd7fa700 1 -- 172.21.15.97:0/3712147646 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92f401d7f0 con 0x7f93000f6510 2024-02-19T22:14:57.742 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:14:57.743 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 msgr2=0x7f92f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 0x7f92f00323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f92ec0096a0 tx=0x7f92ec006b40).stop 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 msgr2=0x7f93001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f92f4005010 tx=0x7f92f400b040).stop 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 shutdown_connections 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92f002ff10 0x7f92f00323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 --2- 172.21.15.97:0/3712147646 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93000f6510 0x7f93001067e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.742+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 >> 172.21.15.97:0/3712147646 conn(0x7f93000f1c70 msgr2=0x7f93000f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:57.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.743+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 shutdown_connections 2024-02-19T22:14:57.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:14:57.743+0000 7f9306f63700 1 -- 172.21.15.97:0/3712147646 wait complete. 2024-02-19T22:14:57.745 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:14:57.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:55.788850+0000 mgr.smithi045.bvlamo (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:58.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:14:57.741959+0000 mon.smithi045 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.97:0/3712147646' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:59.368 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:59.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:59.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:14:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:57.789254+0000 mgr.smithi045.bvlamo (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:14:59.789549+0000 mgr.smithi045.bvlamo (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:03.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:01.789949+0000 mgr.smithi045.bvlamo (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:03.929634+0000 mon.smithi045 (mon.0) 123 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:15:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:03.931728+0000 mon.smithi045 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:15:05.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:03.790221+0000 mgr.smithi045.bvlamo (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:05.790508+0000 mgr.smithi045.bvlamo (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:07.790833+0000 mgr.smithi045.bvlamo (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:10.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 -- 172.21.15.97:0/6037996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 msgr2=0x7f93cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 --2- 172.21.15.97:0/6037996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f93c4009fb0 tx=0x7f93c4009b30).stop 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 -- 172.21.15.97:0/6037996 shutdown_connections 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 --2- 172.21.15.97:0/6037996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 -- 172.21.15.97:0/6037996 >> 172.21.15.97:0/6037996 conn(0x7f93cc0f0b70 msgr2=0x7f93cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 -- 172.21.15.97:0/6037996 shutdown_connections 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.903+0000 7f93d4518700 1 -- 172.21.15.97:0/6037996 wait complete. 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d4518700 1 Processor -- start 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d4518700 1 -- start start 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d4518700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d4518700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc0fb960 con 0x7f93cc0f5410 2024-02-19T22:15:10.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d22b4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d22b4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42204/0 (socket says 172.21.15.97:42204) 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d22b4700 1 -- 172.21.15.97:0/2914004533 learned_addr learned my addr 172.21.15.97:0/2914004533 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.904+0000 7f93d22b4700 1 -- 172.21.15.97:0/2914004533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4005040 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93d22b4700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f93c4005010 tx=0x7f93c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4009320 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc0fbb00 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93c4013380 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93c4014e50 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc0fbf90 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.905+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93c4014420 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.906+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc059100 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.906+0000 7f93caffd700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 0x7f93b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.906+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93c4040f40 con 0x7f93cc0f5410 2024-02-19T22:15:10.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.906+0000 7f93d1ab3700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 0x7f93b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:11.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.906+0000 7f93d1ab3700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 0x7f93b40323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f93c00096a0 tx=0x7f93c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:11.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:10.911+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93c40095f0 con 0x7f93cc0f5410 2024-02-19T22:15:11.119 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.118+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93cc0fafa0 con 0x7f93cc0f5410 2024-02-19T22:15:11.120 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.119+0000 7f93caffd700 1 -- 172.21.15.97:0/2914004533 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93c401db70 con 0x7f93cc0f5410 2024-02-19T22:15:11.123 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:11.123 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 msgr2=0x7f93b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 0x7f93b40323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f93c00096a0 tx=0x7f93c0006b40).stop 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 msgr2=0x7f93cc0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f93c4005010 tx=0x7f93c400b040).stop 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 shutdown_connections 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93b402ff10 0x7f93b40323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 --2- 172.21.15.97:0/2914004533 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93cc0f5410 0x7f93cc0fd4e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 >> 172.21.15.97:0/2914004533 conn(0x7f93cc0f0b70 msgr2=0x7f93cc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:11.124 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 shutdown_connections 2024-02-19T22:15:11.125 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:11.120+0000 7f93d4518700 1 -- 172.21.15.97:0/2914004533 wait complete. 2024-02-19T22:15:11.125 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:11.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:09.791183+0000 mgr.smithi045.bvlamo (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:11.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:11.119755+0000 mon.smithi045 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.97:0/2914004533' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:12.839 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:12.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:11.791559+0000 mgr.smithi045.bvlamo (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3183165720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 msgr2=0x7f79a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3183165720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a40f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f7994009fb0 tx=0x7f7994009b30).stop 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3183165720 shutdown_connections 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3183165720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a40f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3183165720 >> 172.21.15.97:0/3183165720 conn(0x7f79a40f1c70 msgr2=0x7f79a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3183165720 shutdown_connections 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.716+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3183165720 wait complete. 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a8b2b700 1 Processor -- start 2024-02-19T22:15:14.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a8b2b700 1 -- start start 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a8b2b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a8b2b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79a4106cd0 con 0x7f79a40f6510 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42212/0 (socket says 172.21.15.97:42212) 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a259c700 1 -- 172.21.15.97:0/3088862389 learned_addr learned my addr 172.21.15.97:0/3088862389 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a259c700 1 -- 172.21.15.97:0/3088862389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7994005040 con 0x7f79a40f6510 2024-02-19T22:15:14.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.717+0000 7f79a259c700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7994005010 tx=0x7f799400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:14.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7994009320 con 0x7f79a40f6510 2024-02-19T22:15:14.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a4106ed0 con 0x7f79a40f6510 2024-02-19T22:15:14.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7994013380 con 0x7f79a40f6510 2024-02-19T22:15:14.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7994014e50 con 0x7f79a40f6510 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a4107360 con 0x7f79a40f6510 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79940095f0 con 0x7f79a40f6510 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.718+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a4045ad0 con 0x7f79a40f6510 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.719+0000 7f7992ffd700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 0x7f798c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.719+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7994040e90 con 0x7f79a40f6510 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.719+0000 7f79a1d9b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 0x7f798c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:14.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.719+0000 7f79a1d9b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 0x7f798c0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79980096a0 tx=0x7f7998006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:14.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.723+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7994014420 con 0x7f79a40f6510 2024-02-19T22:15:14.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.927+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79a4108210 con 0x7f79a40f6510 2024-02-19T22:15:14.929 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.928+0000 7f7992ffd700 1 -- 172.21.15.97:0/3088862389 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7994010230 con 0x7f79a40f6510 2024-02-19T22:15:14.929 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:14.930 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 msgr2=0x7f798c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 0x7f798c0323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79980096a0 tx=0x7f7998006b40).stop 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 msgr2=0x7f79a41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f7994005010 tx=0x7f799400b040).stop 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 shutdown_connections 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f798c02ff10 0x7f798c0323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 --2- 172.21.15.97:0/3088862389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f79a40f6510 0x7f79a41067e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:14.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 >> 172.21.15.97:0/3088862389 conn(0x7f79a40f1c70 msgr2=0x7f79a40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:14.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 shutdown_connections 2024-02-19T22:15:14.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:14.929+0000 7f79a8b2b700 1 -- 172.21.15.97:0/3088862389 wait complete. 2024-02-19T22:15:14.932 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:13.791866+0000 mgr.smithi045.bvlamo (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:14.928616+0000 mon.smithi045 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.97:0/3088862389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:16.602 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:16.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:17.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:15.792230+0000 mgr.smithi045.bvlamo (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 -- 172.21.15.97:0/4238028976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 msgr2=0x7fd3ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 --2- 172.21.15.97:0/4238028976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac0f6930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fd39c009fb0 tx=0x7fd39c009b30).stop 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 -- 172.21.15.97:0/4238028976 shutdown_connections 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 --2- 172.21.15.97:0/4238028976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac0f6930 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 -- 172.21.15.97:0/4238028976 >> 172.21.15.97:0/4238028976 conn(0x7fd3ac0f1c70 msgr2=0x7fd3ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 -- 172.21.15.97:0/4238028976 shutdown_connections 2024-02-19T22:15:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.521+0000 7fd3b095f700 1 -- 172.21.15.97:0/4238028976 wait complete. 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.522+0000 7fd3b095f700 1 Processor -- start 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.522+0000 7fd3b095f700 1 -- start start 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.522+0000 7fd3b095f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.522+0000 7fd3b095f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ac106cd0 con 0x7fd3ac0f6510 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3aa59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3aa59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41494/0 (socket says 172.21.15.97:41494) 2024-02-19T22:15:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3aa59c700 1 -- 172.21.15.97:0/1750420094 learned_addr learned my addr 172.21.15.97:0/1750420094 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3aa59c700 1 -- 172.21.15.97:0/1750420094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c005040 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3aa59c700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd39c005010 tx=0x7fd39c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39c009320 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd39c013380 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39c014e50 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ac106ed0 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.523+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ac107360 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.524+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd39c0095f0 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.524+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ac045ad0 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.524+0000 7fd39affd700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.524+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd39c040f70 con 0x7fd3ac0f6510 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.525+0000 7fd3a9d9b700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:18.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.525+0000 7fd3a9d9b700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 0x7fd3940323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd3a00096a0 tx=0x7fd3a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:18.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.529+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd39c014420 con 0x7fd3ac0f6510 2024-02-19T22:15:18.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.733+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3ac108210 con 0x7fd3ac0f6510 2024-02-19T22:15:18.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.734+0000 7fd39affd700 1 -- 172.21.15.97:0/1750420094 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd39c01d980 con 0x7fd3ac0f6510 2024-02-19T22:15:18.736 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:18.736 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 msgr2=0x7fd3940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 0x7fd3940323d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fd3a00096a0 tx=0x7fd3a0006b40).stop 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 msgr2=0x7fd3ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fd39c005010 tx=0x7fd39c00b040).stop 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 shutdown_connections 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd39402ff10 0x7fd3940323d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 --2- 172.21.15.97:0/1750420094 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3ac0f6510 0x7fd3ac1067e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 >> 172.21.15.97:0/1750420094 conn(0x7fd3ac0f1c70 msgr2=0x7fd3ac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 shutdown_connections 2024-02-19T22:15:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:18.736+0000 7fd3b095f700 1 -- 172.21.15.97:0/1750420094 wait complete. 2024-02-19T22:15:18.739 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:19.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:17.792599+0000 mgr.smithi045.bvlamo (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:19.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:18.735331+0000 mon.smithi045 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.97:0/1750420094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:20.377 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:20.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:21.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:19.792991+0000 mgr.smithi045.bvlamo (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2146535044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 msgr2=0x7f6ff80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2146535044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff80f55c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f6fec009fb0 tx=0x7f6fec009b30).stop 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2146535044 shutdown_connections 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2146535044 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff80f55c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2146535044 >> 172.21.15.97:0/2146535044 conn(0x7f6ff80f09a0 msgr2=0x7f6ff80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2146535044 shutdown_connections 2024-02-19T22:15:22.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2146535044 wait complete. 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 Processor -- start 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.331+0000 7f6ffdcdd700 1 -- start start 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffdcdd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffdcdd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff8106ac0 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffccdb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffccdb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41520/0 (socket says 172.21.15.97:41520) 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffccdb700 1 -- 172.21.15.97:0/2604664197 learned_addr learned my addr 172.21.15.97:0/2604664197 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffccdb700 1 -- 172.21.15.97:0/2604664197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec005040 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffccdb700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6fec0089f0 tx=0x7f6fec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec009320 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff8106cc0 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fec013380 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.332+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fec014e20 con 0x7f6ff80f51a0 2024-02-19T22:15:22.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.333+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8107150 con 0x7f6ff80f51a0 2024-02-19T22:15:22.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.333+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fec0095f0 con 0x7f6ff80f51a0 2024-02-19T22:15:22.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.333+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff8045ad0 con 0x7f6ff80f51a0 2024-02-19T22:15:22.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.333+0000 7f6ff57fa700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 0x7f6fe80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:22.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.333+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fec040f50 con 0x7f6ff80f51a0 2024-02-19T22:15:22.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.334+0000 7f6ff7fff700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 0x7f6fe80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:22.337 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.334+0000 7f6ff7fff700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 0x7f6fe80323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:22.340 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.338+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fec014420 con 0x7f6ff80f51a0 2024-02-19T22:15:22.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.541+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ff80fad30 con 0x7f6ff80f51a0 2024-02-19T22:15:22.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.542+0000 7f6ff57fa700 1 -- 172.21.15.97:0/2604664197 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fec01d980 con 0x7f6ff80f51a0 2024-02-19T22:15:22.544 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:22.544 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:22.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.543+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 msgr2=0x7f6fe80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.543+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 0x7f6fe80323d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f6fe40096a0 tx=0x7f6fe4006b40).stop 2024-02-19T22:15:22.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.543+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 msgr2=0x7f6ff81065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.543+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f6fec0089f0 tx=0x7f6fec00b040).stop 2024-02-19T22:15:22.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 shutdown_connections 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fe802ff10 0x7f6fe80323d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 --2- 172.21.15.97:0/2604664197 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6ff80f51a0 0x7f6ff81065d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 >> 172.21.15.97:0/2604664197 conn(0x7f6ff80f09a0 msgr2=0x7f6ff805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 shutdown_connections 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:22.544+0000 7f6ffdcdd700 1 -- 172.21.15.97:0/2604664197 wait complete. 2024-02-19T22:15:22.546 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:21.793235+0000 mgr.smithi045.bvlamo (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:22.542966+0000 mon.smithi045 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.97:0/2604664197' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:24.224 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:24.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:25.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:23.793477+0000 mgr.smithi045.bvlamo (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:26.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 -- 172.21.15.97:0/746052539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 msgr2=0x7ff084069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 --2- 172.21.15.97:0/746052539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff084069e40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7ff074009fb0 tx=0x7ff074009b30).stop 2024-02-19T22:15:26.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 -- 172.21.15.97:0/746052539 shutdown_connections 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 --2- 172.21.15.97:0/746052539 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff084069e40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 -- 172.21.15.97:0/746052539 >> 172.21.15.97:0/746052539 conn(0x7ff0840f1c70 msgr2=0x7ff0840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 -- 172.21.15.97:0/746052539 shutdown_connections 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.170+0000 7ff08af59700 1 -- 172.21.15.97:0/746052539 wait complete. 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.171+0000 7ff08af59700 1 Processor -- start 2024-02-19T22:15:26.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.171+0000 7ff08af59700 1 -- start start 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.171+0000 7ff08af59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.171+0000 7ff08af59700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff084108e90 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff088cf5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff088cf5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41536/0 (socket says 172.21.15.97:41536) 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff088cf5700 1 -- 172.21.15.97:0/3948581681 learned_addr learned my addr 172.21.15.97:0/3948581681 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff088cf5700 1 -- 172.21.15.97:0/3948581681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074005040 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff088cf5700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff0740089f0 tx=0x7ff07400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff074009320 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff084109090 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff074013380 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff074014e20 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.172+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff084109520 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.173+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0740263a0 con 0x7ff08406b940 2024-02-19T22:15:26.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.173+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff084045ad0 con 0x7ff08406b940 2024-02-19T22:15:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.173+0000 7ff0817fa700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 0x7ff06c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.173+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff074041250 con 0x7ff08406b940 2024-02-19T22:15:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.173+0000 7ff083fff700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 0x7ff06c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:26.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.174+0000 7ff083fff700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 0x7ff06c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff0780096a0 tx=0x7ff078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:26.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.178+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff07401d470 con 0x7ff08406b940 2024-02-19T22:15:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.382+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff08410a3d0 con 0x7ff08406b940 2024-02-19T22:15:26.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.383+0000 7ff0817fa700 1 -- 172.21.15.97:0/3948581681 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0740126f0 con 0x7ff08406b940 2024-02-19T22:15:26.385 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:26.385 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 msgr2=0x7ff06c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 0x7ff06c0323d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7ff0780096a0 tx=0x7ff078006b40).stop 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 msgr2=0x7ff0841089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff0740089f0 tx=0x7ff07400b040).stop 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 shutdown_connections 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff06c02ff10 0x7ff06c0323d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 --2- 172.21.15.97:0/3948581681 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff08406b940 0x7ff0841089a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 >> 172.21.15.97:0/3948581681 conn(0x7ff0840f1c70 msgr2=0x7ff0840f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 shutdown_connections 2024-02-19T22:15:26.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:26.385+0000 7ff08af59700 1 -- 172.21.15.97:0/3948581681 wait complete. 2024-02-19T22:15:26.388 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:26.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:26.384252+0000 mon.smithi045 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.97:0/3948581681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:27.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:25.793846+0000 mgr.smithi045.bvlamo (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:27.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:27.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:29.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- 172.21.15.97:0/2327071158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 msgr2=0x7fe3780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:29.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 --2- 172.21.15.97:0/2327071158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3780f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fe368009fb0 tx=0x7fe368009b30).stop 2024-02-19T22:15:29.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- 172.21.15.97:0/2327071158 shutdown_connections 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 --2- 172.21.15.97:0/2327071158 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3780f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- 172.21.15.97:0/2327071158 >> 172.21.15.97:0/2327071158 conn(0x7fe3780f0b70 msgr2=0x7fe3780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- 172.21.15.97:0/2327071158 shutdown_connections 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- 172.21.15.97:0/2327071158 wait complete. 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 Processor -- start 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.855+0000 7fe37fb54700 1 -- start start 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37fb54700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37fb54700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe378106cd0 con 0x7fe3780f5410 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37d8f0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37d8f0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48228/0 (socket says 172.21.15.97:48228) 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37d8f0700 1 -- 172.21.15.97:0/1669194465 learned_addr learned my addr 172.21.15.97:0/1669194465 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37d8f0700 1 -- 172.21.15.97:0/1669194465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368005040 con 0x7fe3780f5410 2024-02-19T22:15:29.859 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37d8f0700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe368005010 tx=0x7fe36800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:29.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe368009320 con 0x7fe3780f5410 2024-02-19T22:15:29.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.856+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe378106ed0 con 0x7fe3780f5410 2024-02-19T22:15:29.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe368013380 con 0x7fe3780f5410 2024-02-19T22:15:29.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe368014e50 con 0x7fe3780f5410 2024-02-19T22:15:29.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe378107360 con 0x7fe3780f5410 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe378045ad0 con 0x7fe3780f5410 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe3680095f0 con 0x7fe3780f5410 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.857+0000 7fe36e7fc700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 0x7fe3640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.858+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe368040f70 con 0x7fe3780f5410 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.858+0000 7fe37d0ef700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 0x7fe3640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:29.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.858+0000 7fe37d0ef700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 0x7fe3640323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe3740096a0 tx=0x7fe374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:29.864 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:29.862+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe368014420 con 0x7fe3780f5410 2024-02-19T22:15:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:27.794192+0000 mgr.smithi045.bvlamo (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:30.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.070+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3781081b0 con 0x7fe3780f5410 2024-02-19T22:15:30.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.071+0000 7fe36e7fc700 1 -- 172.21.15.97:0/1669194465 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe368010230 con 0x7fe3780f5410 2024-02-19T22:15:30.072 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:30.073 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 msgr2=0x7fe3640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 0x7fe3640323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe3740096a0 tx=0x7fe374006b40).stop 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 msgr2=0x7fe3781067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe368005010 tx=0x7fe36800b040).stop 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 shutdown_connections 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe36402ff10 0x7fe3640323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 --2- 172.21.15.97:0/1669194465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe3780f5410 0x7fe3781067e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 >> 172.21.15.97:0/1669194465 conn(0x7fe3780f0b70 msgr2=0x7fe3780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 shutdown_connections 2024-02-19T22:15:30.074 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:30.072+0000 7fe37fb54700 1 -- 172.21.15.97:0/1669194465 wait complete. 2024-02-19T22:15:30.075 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:30.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:30.071785+0000 mon.smithi045 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.97:0/1669194465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:31.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:31.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:31.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:29.794578+0000 mgr.smithi045.bvlamo (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- 172.21.15.97:0/1634244774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 msgr2=0x7f09b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 --2- 172.21.15.97:0/1634244774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b00f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f09a4009fb0 tx=0x7f09a4009b30).stop 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- 172.21.15.97:0/1634244774 shutdown_connections 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 --2- 172.21.15.97:0/1634244774 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b00f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- 172.21.15.97:0/1634244774 >> 172.21.15.97:0/1634244774 conn(0x7f09b00f0b70 msgr2=0x7f09b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- 172.21.15.97:0/1634244774 shutdown_connections 2024-02-19T22:15:33.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- 172.21.15.97:0/1634244774 wait complete. 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 Processor -- start 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- start start 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.487+0000 7f09b820d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0106ca0 con 0x7f09b00f5410 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b5fa9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b5fa9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48256/0 (socket says 172.21.15.97:48256) 2024-02-19T22:15:33.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b5fa9700 1 -- 172.21.15.97:0/112153985 learned_addr learned my addr 172.21.15.97:0/112153985 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:33.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b5fa9700 1 -- 172.21.15.97:0/112153985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a4005040 con 0x7f09b00f5410 2024-02-19T22:15:33.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b5fa9700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f09a40089f0 tx=0x7f09a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a4009320 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09a4013380 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0106ea0 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a401ed80 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.488+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0107330 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.489+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09a401e420 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.489+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b0045ad0 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.489+0000 7f09a27fc700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.489+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09a4041110 con 0x7f09b00f5410 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.489+0000 7f09b57a8700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:33.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.490+0000 7f09b57a8700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f09ac0096a0 tx=0x7f09ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:33.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.494+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09a401cb10 con 0x7f09b00f5410 2024-02-19T22:15:33.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.699+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09b00fafa0 con 0x7f09b00f5410 2024-02-19T22:15:33.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.700+0000 7f09a27fc700 1 -- 172.21.15.97:0/112153985 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09a4004490 con 0x7f09b00f5410 2024-02-19T22:15:33.702 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:33.702 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:33.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 msgr2=0x7f099c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 0x7f099c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f09ac0096a0 tx=0x7f09ac006b40).stop 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 msgr2=0x7f09b01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f09a40089f0 tx=0x7f09a400b040).stop 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 shutdown_connections 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f099c02ff60 0x7f099c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 --2- 172.21.15.97:0/112153985 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09b00f5410 0x7f09b01067b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 >> 172.21.15.97:0/112153985 conn(0x7f09b00f0b70 msgr2=0x7f09b005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 shutdown_connections 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:33.702+0000 7f09b820d700 1 -- 172.21.15.97:0/112153985 wait complete. 2024-02-19T22:15:33.704 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:33.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:31.794972+0000 mgr.smithi045.bvlamo (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:34.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:33.701311+0000 mon.smithi045 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.97:0/112153985' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:35.325 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:35.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:35.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:33.795343+0000 mgr.smithi045.bvlamo (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:37.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 -- 172.21.15.97:0/154932507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 msgr2=0x7f67240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 --2- 172.21.15.97:0/154932507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67240f6930 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f671c009fb0 tx=0x7f671c009b30).stop 2024-02-19T22:15:37.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 -- 172.21.15.97:0/154932507 shutdown_connections 2024-02-19T22:15:37.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 --2- 172.21.15.97:0/154932507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67240f6930 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 -- 172.21.15.97:0/154932507 >> 172.21.15.97:0/154932507 conn(0x7f67240f1c70 msgr2=0x7f67240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 -- 172.21.15.97:0/154932507 shutdown_connections 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.281+0000 7f672c0dd700 1 -- 172.21.15.97:0/154932507 wait complete. 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f672c0dd700 1 Processor -- start 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f672c0dd700 1 -- start start 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f672c0dd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f672c0dd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6724106cd0 con 0x7f67240f6510 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f6729e79700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f6729e79700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33620/0 (socket says 172.21.15.97:33620) 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f6729e79700 1 -- 172.21.15.97:0/2249167116 learned_addr learned my addr 172.21.15.97:0/2249167116 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f6729e79700 1 -- 172.21.15.97:0/2249167116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c005040 con 0x7f67240f6510 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.282+0000 7f6729e79700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f671c005010 tx=0x7f671c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f671c009320 con 0x7f67240f6510 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f671c013380 con 0x7f67240f6510 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6724106ed0 con 0x7f67240f6510 2024-02-19T22:15:37.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f671c014e50 con 0x7f67240f6510 2024-02-19T22:15:37.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6724107360 con 0x7f67240f6510 2024-02-19T22:15:37.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f671c0263a0 con 0x7f67240f6510 2024-02-19T22:15:37.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6724045ad0 con 0x7f67240f6510 2024-02-19T22:15:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.283+0000 7f67227fc700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 0x7f670c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.284+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f671c041190 con 0x7f67240f6510 2024-02-19T22:15:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.284+0000 7f6729678700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 0x7f670c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:37.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.284+0000 7f6729678700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 0x7f670c0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:37.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.288+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f671c012510 con 0x7f67240f6510 2024-02-19T22:15:37.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.493+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6724108210 con 0x7f67240f6510 2024-02-19T22:15:37.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.494+0000 7f67227fc700 1 -- 172.21.15.97:0/2249167116 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f671c012330 con 0x7f67240f6510 2024-02-19T22:15:37.495 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:37.496 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:37.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 msgr2=0x7f670c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 0x7f670c0323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f67180096a0 tx=0x7f6718006b40).stop 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 msgr2=0x7f67241067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f671c005010 tx=0x7f671c00b040).stop 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 shutdown_connections 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f670c02ff10 0x7f670c0323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 --2- 172.21.15.97:0/2249167116 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f67240f6510 0x7f67241067e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 >> 172.21.15.97:0/2249167116 conn(0x7f67240f1c70 msgr2=0x7f67240f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 shutdown_connections 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:37.495+0000 7f672c0dd700 1 -- 172.21.15.97:0/2249167116 wait complete. 2024-02-19T22:15:37.498 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:37.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:35.795612+0000 mgr.smithi045.bvlamo (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:37.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:37.494888+0000 mon.smithi045 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.97:0/2249167116' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:39.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:39.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:37.795902+0000 mgr.smithi045.bvlamo (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- 172.21.15.97:0/2418192759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 msgr2=0x7fe0680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 --2- 172.21.15.97:0/2418192759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe0680f55c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fe058009fb0 tx=0x7fe058009b30).stop 2024-02-19T22:15:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- 172.21.15.97:0/2418192759 shutdown_connections 2024-02-19T22:15:41.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 --2- 172.21.15.97:0/2418192759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe0680f55c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- 172.21.15.97:0/2418192759 >> 172.21.15.97:0/2418192759 conn(0x7fe0680f09a0 msgr2=0x7fe0680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- 172.21.15.97:0/2418192759 shutdown_connections 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- 172.21.15.97:0/2418192759 wait complete. 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 Processor -- start 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 -- start start 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.011+0000 7fe06759e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06759e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068060280 con 0x7fe0680f51a0 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33636/0 (socket says 172.21.15.97:33636) 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06659c700 1 -- 172.21.15.97:0/987437684 learned_addr learned my addr 172.21.15.97:0/987437684 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06659c700 1 -- 172.21.15.97:0/987437684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe058005040 con 0x7fe0680f51a0 2024-02-19T22:15:41.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06659c700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe0580089f0 tx=0x7fe05800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe058009320 con 0x7fe0680f51a0 2024-02-19T22:15:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe058013380 con 0x7fe0680f51a0 2024-02-19T22:15:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe068060480 con 0x7fe0680f51a0 2024-02-19T22:15:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe058014e20 con 0x7fe0680f51a0 2024-02-19T22:15:41.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.012+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0681099c0 con 0x7fe0680f51a0 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.013+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0580095f0 con 0x7fe0680f51a0 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.013+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe068045ad0 con 0x7fe0680f51a0 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.013+0000 7fe056ffd700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 0x7fe0500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.013+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe058040e90 con 0x7fe0680f51a0 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.014+0000 7fe065d9b700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 0x7fe0500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:41.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.014+0000 7fe065d9b700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 0x7fe0500323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe05c0096a0 tx=0x7fe05c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:41.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.018+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe058014420 con 0x7fe0680f51a0 2024-02-19T22:15:41.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.222+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0680fad30 con 0x7fe0680f51a0 2024-02-19T22:15:41.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.223+0000 7fe056ffd700 1 -- 172.21.15.97:0/987437684 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe05801d980 con 0x7fe0680f51a0 2024-02-19T22:15:41.225 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:41.225 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 msgr2=0x7fe0500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 0x7fe0500323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fe05c0096a0 tx=0x7fe05c006b40).stop 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 msgr2=0x7fe068109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe0580089f0 tx=0x7fe05800b040).stop 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 shutdown_connections 2024-02-19T22:15:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe05002ff10 0x7fe0500323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.224+0000 7fe06759e700 1 --2- 172.21.15.97:0/987437684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0680f51a0 0x7fe068109290 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.225+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 >> 172.21.15.97:0/987437684 conn(0x7fe0680f09a0 msgr2=0x7fe0680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.225+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 shutdown_connections 2024-02-19T22:15:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:41.225+0000 7fe06759e700 1 -- 172.21.15.97:0/987437684 wait complete. 2024-02-19T22:15:41.227 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:41.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:39.796234+0000 mgr.smithi045.bvlamo (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:41.224052+0000 mon.smithi045 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.97:0/987437684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:43.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:43.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:43.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:41.796572+0000 mgr.smithi045.bvlamo (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- 172.21.15.97:0/1536332859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 msgr2=0x7fdb1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1536332859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f55c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fdb04009fb0 tx=0x7fdb04009b30).stop 2024-02-19T22:15:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- 172.21.15.97:0/1536332859 shutdown_connections 2024-02-19T22:15:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1536332859 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f55c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- 172.21.15.97:0/1536332859 >> 172.21.15.97:0/1536332859 conn(0x7fdb1c0f09a0 msgr2=0x7fdb1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- 172.21.15.97:0/1536332859 shutdown_connections 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- 172.21.15.97:0/1536332859 wait complete. 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 Processor -- start 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.131+0000 7fdb2109a700 1 -- start start 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb2109a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:45.134 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb2109a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c0f13a0 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.135 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33650/0 (socket says 172.21.15.97:33650) 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb1bfff700 1 -- 172.21.15.97:0/1313944334 learned_addr learned my addr 172.21.15.97:0/1313944334 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb1bfff700 1 -- 172.21.15.97:0/1313944334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04005040 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.132+0000 7fdb1bfff700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdb040089f0 tx=0x7fdb0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb04009320 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb04013380 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c0f1540 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb04014e20 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c0f19d0 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb04014420 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1c059100 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.133+0000 7fdb18ff9700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 0x7fdb080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:45.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.134+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb04040f00 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.134+0000 7fdb1b7fe700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 0x7fdb080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:45.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.134+0000 7fdb1b7fe700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 0x7fdb080323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:45.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.139+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb0401db70 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.344+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb1c045ad0 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.344+0000 7fdb18ff9700 1 -- 172.21.15.97:0/1313944334 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdb0401c730 con 0x7fdb1c0f51a0 2024-02-19T22:15:45.347 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:45.347 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 msgr2=0x7fdb080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 0x7fdb080323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).stop 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 msgr2=0x7fdb1c0f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fdb040089f0 tx=0x7fdb0400b040).stop 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 shutdown_connections 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdb0802ff10 0x7fdb080323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 --2- 172.21.15.97:0/1313944334 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f2f20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 >> 172.21.15.97:0/1313944334 conn(0x7fdb1c0f09a0 msgr2=0x7fdb1c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 shutdown_connections 2024-02-19T22:15:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:45.346+0000 7fdb2109a700 1 -- 172.21.15.97:0/1313944334 wait complete. 2024-02-19T22:15:45.349 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:43.796926+0000 mgr.smithi045.bvlamo (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:45.345464+0000 mon.smithi045 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.97:0/1313944334' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:46.940 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:46.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:47.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:45.797231+0000 mgr.smithi045.bvlamo (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- 172.21.15.97:0/3201336185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 msgr2=0x7fad880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 --2- 172.21.15.97:0/3201336185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad880f55c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fad84009fb0 tx=0x7fad84009b30).stop 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- 172.21.15.97:0/3201336185 shutdown_connections 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 --2- 172.21.15.97:0/3201336185 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad880f55c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- 172.21.15.97:0/3201336185 >> 172.21.15.97:0/3201336185 conn(0x7fad880f09a0 msgr2=0x7fad880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- 172.21.15.97:0/3201336185 shutdown_connections 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- 172.21.15.97:0/3201336185 wait complete. 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 Processor -- start 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.725+0000 7fad8e502700 1 -- start start 2024-02-19T22:15:48.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8e502700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8e502700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad88106ad0 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8d500700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8d500700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41354/0 (socket says 172.21.15.97:41354) 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8d500700 1 -- 172.21.15.97:0/1828199575 learned_addr learned my addr 172.21.15.97:0/1828199575 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8d500700 1 -- 172.21.15.97:0/1828199575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84005040 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8d500700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fad840089f0 tx=0x7fad8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad84009320 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad88106cd0 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad84013380 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.726+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad84014e20 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.727+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad88107160 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.727+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad840095f0 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.727+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad88045ad0 con 0x7fad880f51a0 2024-02-19T22:15:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.728+0000 7fad7dffb700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 0x7fad740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:48.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.728+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad84040ec0 con 0x7fad880f51a0 2024-02-19T22:15:48.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.728+0000 7fad8ccff700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 0x7fad740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:48.733 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.728+0000 7fad8ccff700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 0x7fad740323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fad780096a0 tx=0x7fad78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:48.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.733+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad84014420 con 0x7fad880f51a0 2024-02-19T22:15:48.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.938+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad88107fb0 con 0x7fad880f51a0 2024-02-19T22:15:48.940 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.938+0000 7fad7dffb700 1 -- 172.21.15.97:0/1828199575 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad84010230 con 0x7fad880f51a0 2024-02-19T22:15:48.940 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:48.940 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 msgr2=0x7fad740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 0x7fad740323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fad780096a0 tx=0x7fad78006b40).stop 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 msgr2=0x7fad881065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fad840089f0 tx=0x7fad8400b040).stop 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 shutdown_connections 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fad7402ff10 0x7fad740323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 --2- 172.21.15.97:0/1828199575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fad880f51a0 0x7fad881065e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:48.942 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 >> 172.21.15.97:0/1828199575 conn(0x7fad880f09a0 msgr2=0x7fad880f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:48.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 shutdown_connections 2024-02-19T22:15:48.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:48.940+0000 7fad8e502700 1 -- 172.21.15.97:0/1828199575 wait complete. 2024-02-19T22:15:48.943 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:49.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:47.797552+0000 mgr.smithi045.bvlamo (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:48.939416+0000 mon.smithi045 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.97:0/1828199575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:50.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:50.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:51.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:49.797838+0000 mgr.smithi045.bvlamo (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.408+0000 7f987885f700 1 -- 172.21.15.97:0/3900331671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 msgr2=0x7f98740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.408+0000 7f987885f700 1 --2- 172.21.15.97:0/3900331671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98740f55c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f9864009fb0 tx=0x7f9864009b30).stop 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- 172.21.15.97:0/3900331671 shutdown_connections 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 --2- 172.21.15.97:0/3900331671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98740f55c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- 172.21.15.97:0/3900331671 >> 172.21.15.97:0/3900331671 conn(0x7f98740f09a0 msgr2=0x7f98740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- 172.21.15.97:0/3900331671 shutdown_connections 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- 172.21.15.97:0/3900331671 wait complete. 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 Processor -- start 2024-02-19T22:15:52.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- start start 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.409+0000 7f987885f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9874106aa0 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f9872d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f9872d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41372/0 (socket says 172.21.15.97:41372) 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f9872d9d700 1 -- 172.21.15.97:0/1956413546 learned_addr learned my addr 172.21.15.97:0/1956413546 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f9872d9d700 1 -- 172.21.15.97:0/1956413546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864005040 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f9872d9d700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f98640089f0 tx=0x7f986400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9864009320 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9874106ca0 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9864013380 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9864014e20 con 0x7f98740f51a0 2024-02-19T22:15:52.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.410+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9874107130 con 0x7f98740f51a0 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.411+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98640263a0 con 0x7f98740f51a0 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.411+0000 7f985b7fe700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 0x7f985c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.411+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9864040ec0 con 0x7f98740f51a0 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.411+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9874045ad0 con 0x7f98740f51a0 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.411+0000 7f987259c700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 0x7f985c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:52.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.412+0000 7f987259c700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 0x7f985c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f98680096a0 tx=0x7f9868006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:52.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.416+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9864012920 con 0x7f98740f51a0 2024-02-19T22:15:52.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.622+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9874107f80 con 0x7f98740f51a0 2024-02-19T22:15:52.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.622+0000 7f985b7fe700 1 -- 172.21.15.97:0/1956413546 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f986402b020 con 0x7f98740f51a0 2024-02-19T22:15:52.624 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:52.624 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 msgr2=0x7f985c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 0x7f985c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f98680096a0 tx=0x7f9868006b40).stop 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 msgr2=0x7f98741065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f98640089f0 tx=0x7f986400b040).stop 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 shutdown_connections 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f985c02ff10 0x7f985c0323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 --2- 172.21.15.97:0/1956413546 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f98740f51a0 0x7f98741065b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 >> 172.21.15.97:0/1956413546 conn(0x7f98740f09a0 msgr2=0x7f98740f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 shutdown_connections 2024-02-19T22:15:52.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:52.624+0000 7f987885f700 1 -- 172.21.15.97:0/1956413546 wait complete. 2024-02-19T22:15:52.627 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:53.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:51.798153+0000 mgr.smithi045.bvlamo (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:52.623523+0000 mon.smithi045 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.97:0/1956413546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:54.336 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:54.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:55.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:53.798412+0000 mgr.smithi045.bvlamo (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:56.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 -- 172.21.15.97:0/2036581347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 msgr2=0x7fe52c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 --2- 172.21.15.97:0/2036581347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c0f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fe520009fb0 tx=0x7fe520009b30).stop 2024-02-19T22:15:56.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 -- 172.21.15.97:0/2036581347 shutdown_connections 2024-02-19T22:15:56.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 --2- 172.21.15.97:0/2036581347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c0f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 -- 172.21.15.97:0/2036581347 >> 172.21.15.97:0/2036581347 conn(0x7fe52c0f0b70 msgr2=0x7fe52c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 -- 172.21.15.97:0/2036581347 shutdown_connections 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 -- 172.21.15.97:0/2036581347 wait complete. 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.169+0000 7fe530a6f700 1 Processor -- start 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe530a6f700 1 -- start start 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe530a6f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe530a6f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c060240 con 0x7fe52c0f5410 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe52a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe52a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41394/0 (socket says 172.21.15.97:41394) 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe52a59c700 1 -- 172.21.15.97:0/3311956524 learned_addr learned my addr 172.21.15.97:0/3311956524 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe52a59c700 1 -- 172.21.15.97:0/3311956524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe520005040 con 0x7fe52c0f5410 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.170+0000 7fe52a59c700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe520005010 tx=0x7fe52000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe520009320 con 0x7fe52c0f5410 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c060440 con 0x7fe52c0f5410 2024-02-19T22:15:56.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe520013380 con 0x7fe52c0f5410 2024-02-19T22:15:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe520014e50 con 0x7fe52c0f5410 2024-02-19T22:15:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c109b90 con 0x7fe52c0f5410 2024-02-19T22:15:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5200095f0 con 0x7fe52c0f5410 2024-02-19T22:15:56.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c045ad0 con 0x7fe52c0f5410 2024-02-19T22:15:56.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.171+0000 7fe51affd700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 0x7fe5140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:56.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.172+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe520040ed0 con 0x7fe52c0f5410 2024-02-19T22:15:56.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.172+0000 7fe529d9b700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 0x7fe5140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:56.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.172+0000 7fe529d9b700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 0x7fe5140323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe51c0096a0 tx=0x7fe51c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:56.178 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.176+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe520014420 con 0x7fe52c0f5410 2024-02-19T22:15:56.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.381+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe52c10a3b0 con 0x7fe52c0f5410 2024-02-19T22:15:56.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.382+0000 7fe51affd700 1 -- 172.21.15.97:0/3311956524 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe520010230 con 0x7fe52c0f5410 2024-02-19T22:15:56.384 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:15:56.384 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:56.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 msgr2=0x7fe5140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 0x7fe5140323d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fe51c0096a0 tx=0x7fe51c006b40).stop 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 msgr2=0x7fe52c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fe520005010 tx=0x7fe52000b040).stop 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 shutdown_connections 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe51402ff10 0x7fe5140323d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 --2- 172.21.15.97:0/3311956524 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe52c0f5410 0x7fe52c1094e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 >> 172.21.15.97:0/3311956524 conn(0x7fe52c0f0b70 msgr2=0x7fe52c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 shutdown_connections 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:15:56.384+0000 7fe530a6f700 1 -- 172.21.15.97:0/3311956524 wait complete. 2024-02-19T22:15:56.386 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:15:56.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:56 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:15:56.383099+0000 mon.smithi045 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.97:0/3311956524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:57.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:55.798744+0000 mgr.smithi045.bvlamo (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:58.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:58.040 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:59.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:15:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:57.799148+0000 mgr.smithi045.bvlamo (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:00.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 -- 172.21.15.97:0/1143284085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 msgr2=0x7faca80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 --2- 172.21.15.97:0/1143284085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca80f6930 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fac98009fb0 tx=0x7fac98009b30).stop 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 -- 172.21.15.97:0/1143284085 shutdown_connections 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 --2- 172.21.15.97:0/1143284085 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca80f6930 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 -- 172.21.15.97:0/1143284085 >> 172.21.15.97:0/1143284085 conn(0x7faca80f1c70 msgr2=0x7faca80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 -- 172.21.15.97:0/1143284085 shutdown_connections 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.050+0000 7facb0589700 1 -- 172.21.15.97:0/1143284085 wait complete. 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facb0589700 1 Processor -- start 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facb0589700 1 -- start start 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facb0589700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facb0589700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8106cd0 con 0x7faca80f6510 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facae325700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facae325700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56548/0 (socket says 172.21.15.97:56548) 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.051+0000 7facae325700 1 -- 172.21.15.97:0/204483854 learned_addr learned my addr 172.21.15.97:0/204483854 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7facae325700 1 -- 172.21.15.97:0/204483854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac98005040 con 0x7faca80f6510 2024-02-19T22:16:00.054 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7facae325700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fac980089f0 tx=0x7fac9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac98009320 con 0x7faca80f6510 2024-02-19T22:16:00.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca8106ed0 con 0x7faca80f6510 2024-02-19T22:16:00.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac98013380 con 0x7faca80f6510 2024-02-19T22:16:00.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac9801ed80 con 0x7faca80f6510 2024-02-19T22:16:00.055 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.052+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca8107360 con 0x7faca80f6510 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.053+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac9801e420 con 0x7faca80f6510 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.053+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca8045ad0 con 0x7faca80f6510 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.053+0000 7fac9effd700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 0x7fac94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.053+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac980410e0 con 0x7faca80f6510 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.053+0000 7facadb24700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 0x7fac94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.056 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.054+0000 7facadb24700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 0x7fac94032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7faca40096a0 tx=0x7faca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.059 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.058+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac9801cb10 con 0x7faca80f6510 2024-02-19T22:16:00.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.265+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faca8108210 con 0x7faca80f6510 2024-02-19T22:16:00.267 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.266+0000 7fac9effd700 1 -- 172.21.15.97:0/204483854 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac9802c2d0 con 0x7faca80f6510 2024-02-19T22:16:00.268 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:00.268 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:00.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.267+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 msgr2=0x7fac94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.267+0000 7facb0589700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 0x7fac94032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7faca40096a0 tx=0x7faca4006b40).stop 2024-02-19T22:16:00.269 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 msgr2=0x7faca81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fac980089f0 tx=0x7fac9800b040).stop 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 shutdown_connections 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac9402ff60 0x7fac94032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 --2- 172.21.15.97:0/204483854 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faca80f6510 0x7faca81067e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 >> 172.21.15.97:0/204483854 conn(0x7faca80f1c70 msgr2=0x7faca80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 shutdown_connections 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:00.268+0000 7facb0589700 1 -- 172.21.15.97:0/204483854 wait complete. 2024-02-19T22:16:00.270 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:00.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:00.266812+0000 mon.smithi045 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.97:0/204483854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:15:59.799447+0000 mgr.smithi045.bvlamo (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:01.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:01.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:03.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 -- 172.21.15.97:0/2698551074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 msgr2=0x7f2b1c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:03.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 --2- 172.21.15.97:0/2698551074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c0f90f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2b04009fb0 tx=0x7f2b04009b30).stop 2024-02-19T22:16:03.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 -- 172.21.15.97:0/2698551074 shutdown_connections 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 --2- 172.21.15.97:0/2698551074 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c0f90f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 -- 172.21.15.97:0/2698551074 >> 172.21.15.97:0/2698551074 conn(0x7f2b1c0f0b70 msgr2=0x7f2b1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 -- 172.21.15.97:0/2698551074 shutdown_connections 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 -- 172.21.15.97:0/2698551074 wait complete. 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.822+0000 7f2b22487700 1 Processor -- start 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b22487700 1 -- start start 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b22487700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b22487700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b1c106c90 con 0x7f2b1c060280 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56552/0 (socket says 172.21.15.97:56552) 2024-02-19T22:16:03.825 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b1bfff700 1 -- 172.21.15.97:0/993655695 learned_addr learned my addr 172.21.15.97:0/993655695 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b1bfff700 1 -- 172.21.15.97:0/993655695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b04005040 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b1bfff700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2b040089f0 tx=0x7f2b0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b04009320 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1c106e90 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b04013380 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.823+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b04014e20 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.824+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1c107320 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.824+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b040095f0 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.824+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1c045ad0 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.824+0000 7f2b18ff9700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 0x7f2b080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.824+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b04040f50 con 0x7f2b1c060280 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.825+0000 7f2b1b7fe700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 0x7f2b080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.825+0000 7f2b1b7fe700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 0x7f2b080323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2b0c0096a0 tx=0x7f2b0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:03.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:03.829+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b04014420 con 0x7f2b1c060280 2024-02-19T22:16:03.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:01.799722+0000 mgr.smithi045.bvlamo (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:04.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.035+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b1c1081d0 con 0x7f2b1c060280 2024-02-19T22:16:04.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.036+0000 7f2b18ff9700 1 -- 172.21.15.97:0/993655695 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2b0401d980 con 0x7f2b1c060280 2024-02-19T22:16:04.038 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:04.038 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:04.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 msgr2=0x7f2b080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 0x7f2b080323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2b0c0096a0 tx=0x7f2b0c006b40).stop 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 msgr2=0x7f2b1c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2b040089f0 tx=0x7f2b0400b040).stop 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 shutdown_connections 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2b0802ff10 0x7f2b080323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 --2- 172.21.15.97:0/993655695 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2b1c060280 0x7f2b1c1067a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 >> 172.21.15.97:0/993655695 conn(0x7f2b1c0f0b70 msgr2=0x7f2b1c0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 shutdown_connections 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:04.038+0000 7f2b22487700 1 -- 172.21.15.97:0/993655695 wait complete. 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:03.930966+0000 mon.smithi045 (mon.0) 139 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:16:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:03.932624+0000 mon.smithi045 (mon.0) 140 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:16:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:04.037310+0000 mon.smithi045 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.97:0/993655695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:05.658 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:05.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:05.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:03.799966+0000 mgr.smithi045.bvlamo (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.416+0000 7f43831e0700 1 -- 172.21.15.97:0/3399131973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 msgr2=0x7f437c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.416+0000 7f43831e0700 1 --2- 172.21.15.97:0/3399131973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c0f5830 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4370009fb0 tx=0x7f4370009b30).stop 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.416+0000 7f43831e0700 1 -- 172.21.15.97:0/3399131973 shutdown_connections 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.416+0000 7f43831e0700 1 --2- 172.21.15.97:0/3399131973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c0f5830 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.416+0000 7f43831e0700 1 -- 172.21.15.97:0/3399131973 >> 172.21.15.97:0/3399131973 conn(0x7f437c0f0b70 msgr2=0x7f437c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 -- 172.21.15.97:0/3399131973 shutdown_connections 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 -- 172.21.15.97:0/3399131973 wait complete. 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 Processor -- start 2024-02-19T22:16:07.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 -- start start 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.417+0000 7f43831e0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437c106cc0 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f4380f7c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f4380f7c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51018/0 (socket says 172.21.15.97:51018) 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f4380f7c700 1 -- 172.21.15.97:0/2997554006 learned_addr learned my addr 172.21.15.97:0/2997554006 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f4380f7c700 1 -- 172.21.15.97:0/2997554006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370005040 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f4380f7c700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4370005010 tx=0x7f437000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4370009320 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437c106ec0 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4370013380 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4370014e50 con 0x7f437c0f5410 2024-02-19T22:16:07.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.418+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437c107350 con 0x7f437c0f5410 2024-02-19T22:16:07.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.419+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43700095f0 con 0x7f437c0f5410 2024-02-19T22:16:07.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.419+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437c045ad0 con 0x7f437c0f5410 2024-02-19T22:16:07.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.419+0000 7f43797fa700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 0x7f436c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:07.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.419+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4370040fa0 con 0x7f437c0f5410 2024-02-19T22:16:07.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.419+0000 7f437bfff700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 0x7f436c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:07.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.420+0000 7f437bfff700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 0x7f436c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f43680096a0 tx=0x7f4368006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:07.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.424+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4370014420 con 0x7f437c0f5410 2024-02-19T22:16:07.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.629+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f437c108200 con 0x7f437c0f5410 2024-02-19T22:16:07.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.629+0000 7f43797fa700 1 -- 172.21.15.97:0/2997554006 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4370010230 con 0x7f437c0f5410 2024-02-19T22:16:07.631 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:07.631 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 msgr2=0x7f436c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 0x7f436c0323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f43680096a0 tx=0x7f4368006b40).stop 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 msgr2=0x7f437c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4370005010 tx=0x7f437000b040).stop 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 shutdown_connections 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f436c02ff10 0x7f436c0323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 --2- 172.21.15.97:0/2997554006 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f437c0f5410 0x7f437c1067d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 >> 172.21.15.97:0/2997554006 conn(0x7f437c0f0b70 msgr2=0x7f437c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 shutdown_connections 2024-02-19T22:16:07.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:07.631+0000 7f43831e0700 1 -- 172.21.15.97:0/2997554006 wait complete. 2024-02-19T22:16:07.634 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:05.800237+0000 mgr.smithi045.bvlamo (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:08.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:07.630526+0000 mon.smithi045 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.97:0/2997554006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:09.249 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:09.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:07.800602+0000 mgr.smithi045.bvlamo (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- 172.21.15.97:0/2071496785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 msgr2=0x7efe500f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 --2- 172.21.15.97:0/2071496785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f90f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7efe4c009fb0 tx=0x7efe4c009b30).stop 2024-02-19T22:16:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- 172.21.15.97:0/2071496785 shutdown_connections 2024-02-19T22:16:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 --2- 172.21.15.97:0/2071496785 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f90f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.169 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- 172.21.15.97:0/2071496785 >> 172.21.15.97:0/2071496785 conn(0x7efe500f0b70 msgr2=0x7efe500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- 172.21.15.97:0/2071496785 shutdown_connections 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- 172.21.15.97:0/2071496785 wait complete. 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 Processor -- start 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.166+0000 7efe58762700 1 -- start start 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe58762700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe58762700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500f4ff0 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe564fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe564fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51044/0 (socket says 172.21.15.97:51044) 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe564fe700 1 -- 172.21.15.97:0/691104576 learned_addr learned my addr 172.21.15.97:0/691104576 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe564fe700 1 -- 172.21.15.97:0/691104576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c005040 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe564fe700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efe4c0089f0 tx=0x7efe4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4c009320 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe500f5190 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe4c013380 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4c014e20 con 0x7efe50060280 2024-02-19T22:16:11.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.167+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe500f5620 con 0x7efe50060280 2024-02-19T22:16:11.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.168+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe4c0095f0 con 0x7efe50060280 2024-02-19T22:16:11.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.168+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe50059100 con 0x7efe50060280 2024-02-19T22:16:11.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.168+0000 7efe46ffd700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 0x7efe3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.168+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe4c040ee0 con 0x7efe50060280 2024-02-19T22:16:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.169+0000 7efe55cfd700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 0x7efe3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:11.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.169+0000 7efe55cfd700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 0x7efe3c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efe400096a0 tx=0x7efe40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:11.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.173+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe4c01d980 con 0x7efe50060280 2024-02-19T22:16:11.379 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.378+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe500fc090 con 0x7efe50060280 2024-02-19T22:16:11.380 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.379+0000 7efe46ffd700 1 -- 172.21.15.97:0/691104576 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe4c010230 con 0x7efe50060280 2024-02-19T22:16:11.380 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:11.380 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 msgr2=0x7efe3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 0x7efe3c032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efe400096a0 tx=0x7efe40006b40).stop 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 msgr2=0x7efe500f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7efe4c0089f0 tx=0x7efe4c00b040).stop 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 shutdown_connections 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efe3c02ff60 0x7efe3c032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 --2- 172.21.15.97:0/691104576 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efe50060280 0x7efe500f6b70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 >> 172.21.15.97:0/691104576 conn(0x7efe500f0b70 msgr2=0x7efe500f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:11.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 shutdown_connections 2024-02-19T22:16:11.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:11.380+0000 7efe58762700 1 -- 172.21.15.97:0/691104576 wait complete. 2024-02-19T22:16:11.383 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:11.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:09.801002+0000 mgr.smithi045.bvlamo (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:11.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:11.379612+0000 mon.smithi045 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.97:0/691104576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:13.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:13.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:11.801303+0000 mgr.smithi045.bvlamo (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:14.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- 172.21.15.97:0/1125871249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 msgr2=0x7f786c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:14.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 --2- 172.21.15.97:0/1125871249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f785c009fb0 tx=0x7f785c009b30).stop 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- 172.21.15.97:0/1125871249 shutdown_connections 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 --2- 172.21.15.97:0/1125871249 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- 172.21.15.97:0/1125871249 >> 172.21.15.97:0/1125871249 conn(0x7f786c0f0b70 msgr2=0x7f786c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- 172.21.15.97:0/1125871249 shutdown_connections 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- 172.21.15.97:0/1125871249 wait complete. 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 Processor -- start 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- start start 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.961+0000 7f78740bd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c1072f0 con 0x7f786c0f5410 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f7871e59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f7871e59700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51052/0 (socket says 172.21.15.97:51052) 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f7871e59700 1 -- 172.21.15.97:0/2828531347 learned_addr learned my addr 172.21.15.97:0/2828531347 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:14.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f7871e59700 1 -- 172.21.15.97:0/2828531347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c005040 con 0x7f786c0f5410 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f7871e59700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f785c005010 tx=0x7f785c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785c009320 con 0x7f786c0f5410 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c060220 con 0x7f786c0f5410 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f785c013380 con 0x7f786c0f5410 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f785c014e50 con 0x7f786c0f5410 2024-02-19T22:16:14.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.962+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c10bdf0 con 0x7f786c0f5410 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.963+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f785c0095f0 con 0x7f786c0f5410 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.963+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c045ad0 con 0x7f786c0f5410 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.963+0000 7f78627fc700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 0x7f78580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.963+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f785c040e90 con 0x7f786c0f5410 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.964+0000 7f7871658700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 0x7f78580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:14.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.964+0000 7f7871658700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 0x7f78580323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f78680096a0 tx=0x7f7868006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:14.970 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:14.968+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f785c014420 con 0x7f786c0f5410 2024-02-19T22:16:15.176 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.174+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f786c10c5b0 con 0x7f786c0f5410 2024-02-19T22:16:15.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.175+0000 7f78627fc700 1 -- 172.21.15.97:0/2828531347 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f785c010230 con 0x7f786c0f5410 2024-02-19T22:16:15.177 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:15.177 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.176+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 msgr2=0x7f78580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 0x7f78580323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f78680096a0 tx=0x7f7868006b40).stop 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 msgr2=0x7f786c10b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f785c005010 tx=0x7f785c00b040).stop 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 shutdown_connections 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f785802ff10 0x7f78580323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 --2- 172.21.15.97:0/2828531347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f786c0f5410 0x7f786c10b900 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 >> 172.21.15.97:0/2828531347 conn(0x7f786c0f0b70 msgr2=0x7f786c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 shutdown_connections 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:15.177+0000 7f78740bd700 1 -- 172.21.15.97:0/2828531347 wait complete. 2024-02-19T22:16:15.179 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:13.801719+0000 mgr.smithi045.bvlamo (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:15.176272+0000 mon.smithi045 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.97:0/2828531347' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:16.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:16.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:17.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:15.802020+0000 mgr.smithi045.bvlamo (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.778+0000 7f07167bc700 1 -- 172.21.15.97:0/3908450507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 msgr2=0x7f07100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.778+0000 7f07167bc700 1 --2- 172.21.15.97:0/3908450507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07100f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f0700009fb0 tx=0x7f0700009b30).stop 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.778+0000 7f07167bc700 1 -- 172.21.15.97:0/3908450507 shutdown_connections 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.778+0000 7f07167bc700 1 --2- 172.21.15.97:0/3908450507 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07100f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.778+0000 7f07167bc700 1 -- 172.21.15.97:0/3908450507 >> 172.21.15.97:0/3908450507 conn(0x7f07100f1c70 msgr2=0x7f07100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 -- 172.21.15.97:0/3908450507 shutdown_connections 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 -- 172.21.15.97:0/3908450507 wait complete. 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 Processor -- start 2024-02-19T22:16:18.782 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 -- start start 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.779+0000 7f07167bc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0710106cd0 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:45836/0 (socket says 172.21.15.97:45836) 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070ffff700 1 -- 172.21.15.97:0/2820542446 learned_addr learned my addr 172.21.15.97:0/2820542446 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070ffff700 1 -- 172.21.15.97:0/2820542446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700005040 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070ffff700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0700005010 tx=0x7f070000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0700009320 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0710106ed0 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0700013380 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0700014e50 con 0x7f07100f6510 2024-02-19T22:16:18.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.780+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0710107360 con 0x7f07100f6510 2024-02-19T22:16:18.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.781+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07000095f0 con 0x7f07100f6510 2024-02-19T22:16:18.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.781+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0710045ad0 con 0x7f07100f6510 2024-02-19T22:16:18.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.781+0000 7f070cff9700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:18.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.781+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0700040fa0 con 0x7f07100f6510 2024-02-19T22:16:18.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.781+0000 7f070f7fe700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:18.785 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.782+0000 7f070f7fe700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 0x7f06f80323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:18.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.786+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0700014420 con 0x7f07100f6510 2024-02-19T22:16:18.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:17.802340+0000 mgr.smithi045.bvlamo (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.991+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0710108210 con 0x7f07100f6510 2024-02-19T22:16:18.993 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.992+0000 7f070cff9700 1 -- 172.21.15.97:0/2820542446 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0700010230 con 0x7f07100f6510 2024-02-19T22:16:18.993 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:18.993 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 msgr2=0x7f06f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 0x7f06f80323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f07040096a0 tx=0x7f0704006b40).stop 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 msgr2=0x7f07101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f0700005010 tx=0x7f070000b040).stop 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 shutdown_connections 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f06f802ff10 0x7f06f80323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 --2- 172.21.15.97:0/2820542446 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07100f6510 0x7f07101067e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 >> 172.21.15.97:0/2820542446 conn(0x7f07100f1c70 msgr2=0x7f07100f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:18.995 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 shutdown_connections 2024-02-19T22:16:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:18.993+0000 7f07167bc700 1 -- 172.21.15.97:0/2820542446 wait complete. 2024-02-19T22:16:18.996 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:19.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:18.992649+0000 mon.smithi045 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.97:0/2820542446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:20.688 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:20.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:21.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:19.802625+0000 mgr.smithi045.bvlamo (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:22.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 -- 172.21.15.97:0/386237470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 msgr2=0x7fa4f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 --2- 172.21.15.97:0/386237470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f00f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa4e0009fb0 tx=0x7fa4e0009b30).stop 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 -- 172.21.15.97:0/386237470 shutdown_connections 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 --2- 172.21.15.97:0/386237470 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f00f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 -- 172.21.15.97:0/386237470 >> 172.21.15.97:0/386237470 conn(0x7fa4f00f1c70 msgr2=0x7fa4f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 -- 172.21.15.97:0/386237470 shutdown_connections 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 -- 172.21.15.97:0/386237470 wait complete. 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.522+0000 7fa4f798f700 1 Processor -- start 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f798f700 1 -- start start 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f798f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f798f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f0108ef0 con 0x7fa4f00f6510 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f572b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f572b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:45866/0 (socket says 172.21.15.97:45866) 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f572b700 1 -- 172.21.15.97:0/3642837746 learned_addr learned my addr 172.21.15.97:0/3642837746 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:22.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f572b700 1 -- 172.21.15.97:0/3642837746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e0005040 con 0x7fa4f00f6510 2024-02-19T22:16:22.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.523+0000 7fa4f572b700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa4e0005010 tx=0x7fa4e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:22.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.524+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e0009320 con 0x7fa4f00f6510 2024-02-19T22:16:22.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.524+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f01090f0 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.524+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f0109580 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.524+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4e0013380 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.524+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e0014df0 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4e001e420 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d0004fa0 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4e5ffb700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 0x7fa4dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4e0041190 con 0x7fa4f00f6510 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4f4f2a700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 0x7fa4dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:22.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.525+0000 7fa4f4f2a700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 0x7fa4dc0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa4ec0096a0 tx=0x7fa4ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:22.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.530+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4e001c500 con 0x7fa4f00f6510 2024-02-19T22:16:22.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.732+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4d0005670 con 0x7fa4f00f6510 2024-02-19T22:16:22.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.733+0000 7fa4e5ffb700 1 -- 172.21.15.97:0/3642837746 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4e001c320 con 0x7fa4f00f6510 2024-02-19T22:16:22.735 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:22.735 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:22.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.734+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 msgr2=0x7fa4dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.734+0000 7fa4f798f700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 0x7fa4dc0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa4ec0096a0 tx=0x7fa4ec006b40).stop 2024-02-19T22:16:22.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.734+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 msgr2=0x7fa4f0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.734+0000 7fa4f798f700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa4e0005010 tx=0x7fa4e000b040).stop 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 shutdown_connections 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4dc02ff10 0x7fa4dc0323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 --2- 172.21.15.97:0/3642837746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4f00f6510 0x7fa4f0108a00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 >> 172.21.15.97:0/3642837746 conn(0x7fa4f00f1c70 msgr2=0x7fa4f00f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 shutdown_connections 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:22.735+0000 7fa4f798f700 1 -- 172.21.15.97:0/3642837746 wait complete. 2024-02-19T22:16:22.737 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:21.802942+0000 mgr.smithi045.bvlamo (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:22.733858+0000 mon.smithi045 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.97:0/3642837746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:24.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:24.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:25.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:23.803195+0000 mgr.smithi045.bvlamo (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:26.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 -- 172.21.15.97:0/2632857889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 msgr2=0x7f8cd00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/2632857889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd00f4be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8cc4009fb0 tx=0x7f8cc4009b30).stop 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 -- 172.21.15.97:0/2632857889 shutdown_connections 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/2632857889 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd00f4be0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 -- 172.21.15.97:0/2632857889 >> 172.21.15.97:0/2632857889 conn(0x7f8cd00f02e0 msgr2=0x7f8cd00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 -- 172.21.15.97:0/2632857889 shutdown_connections 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.165+0000 7f8cd66c7700 1 -- 172.21.15.97:0/2632857889 wait complete. 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8cd66c7700 1 Processor -- start 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8cd66c7700 1 -- start start 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8cd66c7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8cd66c7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0106cc0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8ccffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8ccffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:45880/0 (socket says 172.21.15.97:45880) 2024-02-19T22:16:26.190 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.166+0000 7f8ccffff700 1 -- 172.21.15.97:0/3101645344 learned_addr learned my addr 172.21.15.97:0/3101645344 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8ccffff700 1 -- 172.21.15.97:0/3101645344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4005040 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8ccffff700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8cc4005010 tx=0x7f8cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4009320 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd0106ec0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8cc4013380 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0107350 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.167+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc4014dd0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.168+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc40273f0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.168+0000 7f8cccff9700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 0x7f8cb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.168+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc4040e30 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.168+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd0045ad0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.168+0000 7f8ccf7fe700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 0x7f8cb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.169+0000 7f8ccf7fe700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 0x7f8cb8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.173+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc40263a0 con 0x7f8cd00f47c0 2024-02-19T22:16:26.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.379+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cd0108070 con 0x7f8cd00f47c0 2024-02-19T22:16:26.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.380+0000 7f8cccff9700 1 -- 172.21.15.97:0/3101645344 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cc4012330 con 0x7f8cd00f47c0 2024-02-19T22:16:26.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:26.382 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 msgr2=0x7f8cb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 0x7f8cb8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8cc00096a0 tx=0x7f8cc0006b40).stop 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 msgr2=0x7f8cd01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8cc4005010 tx=0x7f8cc400b040).stop 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 shutdown_connections 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8cb802ff60 0x7f8cb8032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 --2- 172.21.15.97:0/3101645344 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8cd00f47c0 0x7f8cd01067d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 >> 172.21.15.97:0/3101645344 conn(0x7f8cd00f02e0 msgr2=0x7f8cd00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 shutdown_connections 2024-02-19T22:16:26.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:26.382+0000 7f8cd66c7700 1 -- 172.21.15.97:0/3101645344 wait complete. 2024-02-19T22:16:26.385 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:26.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:26.381188+0000 mon.smithi045 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.97:0/3101645344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:27.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:25.803473+0000 mgr.smithi045.bvlamo (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:28.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:28.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:29.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 -- 172.21.15.97:0/1741876894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 msgr2=0x7f87200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 --2- 172.21.15.97:0/1741876894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87200f6930 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f8708009fb0 tx=0x7f8708009b30).stop 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 -- 172.21.15.97:0/1741876894 shutdown_connections 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 --2- 172.21.15.97:0/1741876894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87200f6930 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 -- 172.21.15.97:0/1741876894 >> 172.21.15.97:0/1741876894 conn(0x7f87200f1c70 msgr2=0x7f87200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 -- 172.21.15.97:0/1741876894 shutdown_connections 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.847+0000 7f87252a9700 1 -- 172.21.15.97:0/1741876894 wait complete. 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f87252a9700 1 Processor -- start 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f87252a9700 1 -- start start 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f87252a9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f87252a9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8720106cd0 con 0x7f87200f6510 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f871effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:29.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f871effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39264/0 (socket says 172.21.15.97:39264) 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f871effd700 1 -- 172.21.15.97:0/2536869269 learned_addr learned my addr 172.21.15.97:0/2536869269 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f871effd700 1 -- 172.21.15.97:0/2536869269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8708005040 con 0x7f87200f6510 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.848+0000 7f871effd700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f8708005010 tx=0x7f870800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8708009320 con 0x7f87200f6510 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8720106ed0 con 0x7f87200f6510 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8708013380 con 0x7f87200f6510 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8708014e50 con 0x7f87200f6510 2024-02-19T22:16:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8720107360 con 0x7f87200f6510 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.849+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87080095f0 con 0x7f87200f6510 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.850+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8720045ad0 con 0x7f87200f6510 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.850+0000 7f87177fe700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 0x7f870c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.850+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8708040f10 con 0x7f87200f6510 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.850+0000 7f871e7fc700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 0x7f870c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:29.853 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.850+0000 7f871e7fc700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 0x7f870c0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f87100096a0 tx=0x7f8710006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:29.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:29.854+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8708014420 con 0x7f87200f6510 2024-02-19T22:16:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:27.803824+0000 mgr.smithi045.bvlamo (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.061+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8720108210 con 0x7f87200f6510 2024-02-19T22:16:30.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.062+0000 7f87177fe700 1 -- 172.21.15.97:0/2536869269 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8708010230 con 0x7f87200f6510 2024-02-19T22:16:30.064 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:30.064 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 msgr2=0x7f870c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 0x7f870c0323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f87100096a0 tx=0x7f8710006b40).stop 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 msgr2=0x7f87201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f8708005010 tx=0x7f870800b040).stop 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 shutdown_connections 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f870c02ff10 0x7f870c0323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 --2- 172.21.15.97:0/2536869269 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87200f6510 0x7f87201067e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 >> 172.21.15.97:0/2536869269 conn(0x7f87200f1c70 msgr2=0x7f87200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 shutdown_connections 2024-02-19T22:16:30.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:30.064+0000 7f87252a9700 1 -- 172.21.15.97:0/2536869269 wait complete. 2024-02-19T22:16:30.067 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:30.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:30.063093+0000 mon.smithi045 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.97:0/2536869269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:31.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:31.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:31.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:29.804172+0000 mgr.smithi045.bvlamo (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:33.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 -- 172.21.15.97:0/4207561755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 msgr2=0x7fcbf80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 --2- 172.21.15.97:0/4207561755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf80f4be0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fcbec009fb0 tx=0x7fcbec009b30).stop 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 -- 172.21.15.97:0/4207561755 shutdown_connections 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 --2- 172.21.15.97:0/4207561755 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf80f4be0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 -- 172.21.15.97:0/4207561755 >> 172.21.15.97:0/4207561755 conn(0x7fcbf80f02e0 msgr2=0x7fcbf80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 -- 172.21.15.97:0/4207561755 shutdown_connections 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.742+0000 7fcbfd226700 1 -- 172.21.15.97:0/4207561755 wait complete. 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbfd226700 1 Processor -- start 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbfd226700 1 -- start start 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbfd226700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:33.746 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbfd226700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8106cc0 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbf6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbf6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39294/0 (socket says 172.21.15.97:39294) 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbf6d9d700 1 -- 172.21.15.97:0/1068226829 learned_addr learned my addr 172.21.15.97:0/1068226829 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbf6d9d700 1 -- 172.21.15.97:0/1068226829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec005040 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.743+0000 7fcbf6d9d700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcbec005010 tx=0x7fcbec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbec009320 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbec013380 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf8106ec0 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbec014e50 con 0x7fcbf80f47c0 2024-02-19T22:16:33.747 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf8107350 con 0x7fcbf80f47c0 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbec0095f0 con 0x7fcbf80f47c0 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.744+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8045ad0 con 0x7fcbf80f47c0 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.745+0000 7fcbdf7fe700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 0x7fcbe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.745+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbec040f80 con 0x7fcbf80f47c0 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.745+0000 7fcbf659c700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 0x7fcbe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:33.748 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.745+0000 7fcbf659c700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 0x7fcbe00323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcbe80096a0 tx=0x7fcbe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:33.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.749+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbec014420 con 0x7fcbf80f47c0 2024-02-19T22:16:33.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:31.804418+0000 mgr.smithi045.bvlamo (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:33.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.960+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbf8108200 con 0x7fcbf80f47c0 2024-02-19T22:16:33.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.960+0000 7fcbdf7fe700 1 -- 172.21.15.97:0/1068226829 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcbec010230 con 0x7fcbf80f47c0 2024-02-19T22:16:33.963 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:33.963 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:33.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 msgr2=0x7fcbe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:33.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 0x7fcbe00323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcbe80096a0 tx=0x7fcbe8006b40).stop 2024-02-19T22:16:33.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 msgr2=0x7fcbf81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcbec005010 tx=0x7fcbec00b040).stop 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 shutdown_connections 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcbe002ff10 0x7fcbe00323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 --2- 172.21.15.97:0/1068226829 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcbf80f47c0 0x7fcbf81067d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 >> 172.21.15.97:0/1068226829 conn(0x7fcbf80f02e0 msgr2=0x7fcbf80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 shutdown_connections 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:33.962+0000 7fcbfd226700 1 -- 172.21.15.97:0/1068226829 wait complete. 2024-02-19T22:16:33.965 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:34.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:33.961417+0000 mon.smithi045 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.97:0/1068226829' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:35.601 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:35.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:35.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:33.804712+0000 mgr.smithi045.bvlamo (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 -- 172.21.15.97:0/3884869247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 msgr2=0x7fcad80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 --2- 172.21.15.97:0/3884869247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad80f6930 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fcac8009fb0 tx=0x7fcac8009b30).stop 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 -- 172.21.15.97:0/3884869247 shutdown_connections 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 --2- 172.21.15.97:0/3884869247 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad80f6930 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 -- 172.21.15.97:0/3884869247 >> 172.21.15.97:0/3884869247 conn(0x7fcad80f1c70 msgr2=0x7fcad80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:37.594 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 -- 172.21.15.97:0/3884869247 shutdown_connections 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.590+0000 7fcadd330700 1 -- 172.21.15.97:0/3884869247 wait complete. 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcadd330700 1 Processor -- start 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcadd330700 1 -- start start 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcadd330700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcadd330700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8069e20 con 0x7fcad80f6510 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcad6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcad6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39340/0 (socket says 172.21.15.97:39340) 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.591+0000 7fcad6ffd700 1 -- 172.21.15.97:0/4290129575 learned_addr learned my addr 172.21.15.97:0/4290129575 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcad6ffd700 1 -- 172.21.15.97:0/4290129575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8005040 con 0x7fcad80f6510 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcad6ffd700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcac8005010 tx=0x7fcac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8009320 con 0x7fcad80f6510 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcac8013380 con 0x7fcad80f6510 2024-02-19T22:16:37.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad8069fc0 con 0x7fcad80f6510 2024-02-19T22:16:37.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcac8014e50 con 0x7fcad80f6510 2024-02-19T22:16:37.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.592+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad806a450 con 0x7fcad80f6510 2024-02-19T22:16:37.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcac80095f0 con 0x7fcad80f6510 2024-02-19T22:16:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8059100 con 0x7fcad80f6510 2024-02-19T22:16:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcacf7fe700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 0x7fcac40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcac8040f70 con 0x7fcad80f6510 2024-02-19T22:16:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcad67fc700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 0x7fcac40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:37.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.593+0000 7fcad67fc700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 0x7fcac40323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcac00096a0 tx=0x7fcac0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:37.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.598+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcac8014420 con 0x7fcad80f6510 2024-02-19T22:16:37.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.804+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad80fc0a0 con 0x7fcad80f6510 2024-02-19T22:16:37.806 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.805+0000 7fcacf7fe700 1 -- 172.21.15.97:0/4290129575 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcac801d980 con 0x7fcad80f6510 2024-02-19T22:16:37.807 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:37.807 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 msgr2=0x7fcac40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 0x7fcac40323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fcac00096a0 tx=0x7fcac0006b40).stop 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 msgr2=0x7fcad806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fcac8005010 tx=0x7fcac800b040).stop 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 shutdown_connections 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcac402ff10 0x7fcac40323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 --2- 172.21.15.97:0/4290129575 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 >> 172.21.15.97:0/4290129575 conn(0x7fcad80f1c70 msgr2=0x7fcad80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:37.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 shutdown_connections 2024-02-19T22:16:37.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:37.806+0000 7fcadd330700 1 -- 172.21.15.97:0/4290129575 wait complete. 2024-02-19T22:16:37.809 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:37.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:35.805097+0000 mgr.smithi045.bvlamo (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:38.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:37.805791+0000 mon.smithi045 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.97:0/4290129575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:39.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:39.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:37.805416+0000 mgr.smithi045.bvlamo (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:41.374 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.355+0000 7f97508c4700 1 -- 172.21.15.97:0/428487245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 msgr2=0x7f974c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.355+0000 7f97508c4700 1 --2- 172.21.15.97:0/428487245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c0f5830 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f973c009fb0 tx=0x7f973c009b30).stop 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.355+0000 7f97508c4700 1 -- 172.21.15.97:0/428487245 shutdown_connections 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 --2- 172.21.15.97:0/428487245 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c0f5830 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 -- 172.21.15.97:0/428487245 >> 172.21.15.97:0/428487245 conn(0x7f974c0f0b70 msgr2=0x7f974c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 -- 172.21.15.97:0/428487245 shutdown_connections 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 -- 172.21.15.97:0/428487245 wait complete. 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 Processor -- start 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 -- start start 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f97508c4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974c105bd0 con 0x7f974c0f5410 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f974a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:41.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f974a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39358/0 (socket says 172.21.15.97:39358) 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.356+0000 7f974a59c700 1 -- 172.21.15.97:0/1704012736 learned_addr learned my addr 172.21.15.97:0/1704012736 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f974a59c700 1 -- 172.21.15.97:0/1704012736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973c005040 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f974a59c700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f973c0089f0 tx=0x7f973c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f973c009320 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f974c105dd0 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f973c013380 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f973c01ed80 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.357+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974c106260 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.358+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974c045ad0 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.358+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f973c01e420 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.358+0000 7f9742ffd700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 0x7f9738032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.358+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f973c040fc0 con 0x7f974c0f5410 2024-02-19T22:16:41.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.358+0000 7f9749d9b700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 0x7f9738032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:41.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.359+0000 7f9749d9b700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 0x7f9738032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f97340096a0 tx=0x7f9734006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:41.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.363+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f973c0095f0 con 0x7f974c0f5410 2024-02-19T22:16:41.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.566+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f974c1070b0 con 0x7f974c0f5410 2024-02-19T22:16:41.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.567+0000 7f9742ffd700 1 -- 172.21.15.97:0/1704012736 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f973c01cb10 con 0x7f974c0f5410 2024-02-19T22:16:41.569 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:41.569 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 msgr2=0x7f9738032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 0x7f9738032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f97340096a0 tx=0x7f9734006b40).stop 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 msgr2=0x7f974c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f973c0089f0 tx=0x7f973c00b040).stop 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 shutdown_connections 2024-02-19T22:16:41.571 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f973802ff60 0x7f9738032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 --2- 172.21.15.97:0/1704012736 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f974c0f5410 0x7f974c1056e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 >> 172.21.15.97:0/1704012736 conn(0x7f974c0f0b70 msgr2=0x7f974c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:41.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 shutdown_connections 2024-02-19T22:16:41.572 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:41.569+0000 7f97508c4700 1 -- 172.21.15.97:0/1704012736 wait complete. 2024-02-19T22:16:41.572 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:39.805705+0000 mgr.smithi045.bvlamo (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:42.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:41.568178+0000 mon.smithi045 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.97:0/1704012736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:43.218 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:43.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:43.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:41.806097+0000 mgr.smithi045.bvlamo (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 -- 172.21.15.97:0/2500744540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 msgr2=0x7fe8b00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2500744540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b00f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fe8a4009fb0 tx=0x7fe8a4009b30).stop 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 -- 172.21.15.97:0/2500744540 shutdown_connections 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2500744540 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b00f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 -- 172.21.15.97:0/2500744540 >> 172.21.15.97:0/2500744540 conn(0x7fe8b00f1c70 msgr2=0x7fe8b00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 -- 172.21.15.97:0/2500744540 shutdown_connections 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.160+0000 7fe8b613b700 1 -- 172.21.15.97:0/2500744540 wait complete. 2024-02-19T22:16:45.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8b613b700 1 Processor -- start 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8b613b700 1 -- start start 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8b613b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8b613b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b0106cd0 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8af7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8af7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39380/0 (socket says 172.21.15.97:39380) 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.161+0000 7fe8af7fe700 1 -- 172.21.15.97:0/2926933274 learned_addr learned my addr 172.21.15.97:0/2926933274 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe8af7fe700 1 -- 172.21.15.97:0/2926933274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a4005040 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe8af7fe700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe8a4005010 tx=0x7fe8a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8a4009320 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b0106ed0 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8a4013380 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8a4014e50 con 0x7fe8b00f6510 2024-02-19T22:16:45.166 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.162+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b0107360 con 0x7fe8b00f6510 2024-02-19T22:16:45.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.163+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8a40263a0 con 0x7fe8b00f6510 2024-02-19T22:16:45.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.163+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b0045ad0 con 0x7fe8b00f6510 2024-02-19T22:16:45.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.163+0000 7fe897fff700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:45.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.163+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8a4041250 con 0x7fe8b00f6510 2024-02-19T22:16:45.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.163+0000 7fe8aeffd700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:45.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.164+0000 7fe8aeffd700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 0x7fe8980323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:45.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.168+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8a4026e60 con 0x7fe8b00f6510 2024-02-19T22:16:45.375 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.372+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8b0108210 con 0x7fe8b00f6510 2024-02-19T22:16:45.376 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.373+0000 7fe897fff700 1 -- 172.21.15.97:0/2926933274 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8a4012330 con 0x7fe8b00f6510 2024-02-19T22:16:45.376 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:45.376 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:45.377 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 msgr2=0x7fe8980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 0x7fe8980323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fe8a00096a0 tx=0x7fe8a0006b40).stop 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 msgr2=0x7fe8b01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe8a4005010 tx=0x7fe8a400b040).stop 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 shutdown_connections 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe89802ff10 0x7fe8980323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 --2- 172.21.15.97:0/2926933274 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe8b00f6510 0x7fe8b01067e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 >> 172.21.15.97:0/2926933274 conn(0x7fe8b00f1c70 msgr2=0x7fe8b00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 shutdown_connections 2024-02-19T22:16:45.378 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:45.376+0000 7fe8b613b700 1 -- 172.21.15.97:0/2926933274 wait complete. 2024-02-19T22:16:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:45.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:43.806481+0000 mgr.smithi045.bvlamo (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:45.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:45.375213+0000 mon.smithi045 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.97:0/2926933274' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:47.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:47.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:47.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:45.806839+0000 mgr.smithi045.bvlamo (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.888+0000 7fcb3b842700 1 -- 172.21.15.97:0/1770292447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 msgr2=0x7fcb340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.888+0000 7fcb3b842700 1 --2- 172.21.15.97:0/1770292447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb340f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fcb24009fb0 tx=0x7fcb24009b30).stop 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- 172.21.15.97:0/1770292447 shutdown_connections 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 --2- 172.21.15.97:0/1770292447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb340f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- 172.21.15.97:0/1770292447 >> 172.21.15.97:0/1770292447 conn(0x7fcb340f1c70 msgr2=0x7fcb340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- 172.21.15.97:0/1770292447 shutdown_connections 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- 172.21.15.97:0/1770292447 wait complete. 2024-02-19T22:16:48.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 Processor -- start 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- start start 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.889+0000 7fcb3b842700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb34106d00 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb395de700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb395de700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42146/0 (socket says 172.21.15.97:42146) 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb395de700 1 -- 172.21.15.97:0/989536746 learned_addr learned my addr 172.21.15.97:0/989536746 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb395de700 1 -- 172.21.15.97:0/989536746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb24005040 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb395de700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcb24005010 tx=0x7fcb2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb24009320 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb34106f00 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcb24013380 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb24014e50 con 0x7fcb340f6510 2024-02-19T22:16:48.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.890+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb34107390 con 0x7fcb340f6510 2024-02-19T22:16:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.891+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb24014420 con 0x7fcb340f6510 2024-02-19T22:16:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.891+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb34045ad0 con 0x7fcb340f6510 2024-02-19T22:16:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.891+0000 7fcb29ffb700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 0x7fcb200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:48.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.891+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb24040f70 con 0x7fcb340f6510 2024-02-19T22:16:48.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.892+0000 7fcb38ddd700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 0x7fcb200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:48.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.892+0000 7fcb38ddd700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 0x7fcb200323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fcb300096a0 tx=0x7fcb30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:48.898 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:48.896+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb240095f0 con 0x7fcb340f6510 2024-02-19T22:16:49.106 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.105+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb34108240 con 0x7fcb340f6510 2024-02-19T22:16:49.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.105+0000 7fcb29ffb700 1 -- 172.21.15.97:0/989536746 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb2401db70 con 0x7fcb340f6510 2024-02-19T22:16:49.107 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:49.107 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 msgr2=0x7fcb200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 0x7fcb200323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fcb300096a0 tx=0x7fcb30006b40).stop 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 msgr2=0x7fcb34106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcb24005010 tx=0x7fcb2400b040).stop 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 shutdown_connections 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcb2002ff10 0x7fcb200323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 --2- 172.21.15.97:0/989536746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb340f6510 0x7fcb34106810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 >> 172.21.15.97:0/989536746 conn(0x7fcb340f1c70 msgr2=0x7fcb340f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 shutdown_connections 2024-02-19T22:16:49.109 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:49.107+0000 7fcb3b842700 1 -- 172.21.15.97:0/989536746 wait complete. 2024-02-19T22:16:49.110 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:49.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:47.807208+0000 mgr.smithi045.bvlamo (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:49.106392+0000 mon.smithi045 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.97:0/989536746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:50.723 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:50.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:51.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:49.807524+0000 mgr.smithi045.bvlamo (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 -- 172.21.15.97:0/1770090198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 msgr2=0x7fa0700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 --2- 172.21.15.97:0/1770090198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0700f6930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fa064009fb0 tx=0x7fa064009b30).stop 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 -- 172.21.15.97:0/1770090198 shutdown_connections 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 --2- 172.21.15.97:0/1770090198 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0700f6930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 -- 172.21.15.97:0/1770090198 >> 172.21.15.97:0/1770090198 conn(0x7fa0700f1c70 msgr2=0x7fa0700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 -- 172.21.15.97:0/1770090198 shutdown_connections 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.657+0000 7fa077e50700 1 -- 172.21.15.97:0/1770090198 wait complete. 2024-02-19T22:16:52.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa077e50700 1 Processor -- start 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa077e50700 1 -- start start 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa077e50700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa077e50700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070106cd0 con 0x7fa0700f6510 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa075bec700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa075bec700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42156/0 (socket says 172.21.15.97:42156) 2024-02-19T22:16:52.661 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.658+0000 7fa075bec700 1 -- 172.21.15.97:0/182155807 learned_addr learned my addr 172.21.15.97:0/182155807 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:52.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa075bec700 1 -- 172.21.15.97:0/182155807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064005040 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa075bec700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa064005010 tx=0x7fa06400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa064009320 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070106ed0 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa064013380 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa064014e50 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.659+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa070107360 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.660+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0640095f0 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.660+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa070045ad0 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.660+0000 7fa0627fc700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 0x7fa05c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.660+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa064040e90 con 0x7fa0700f6510 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.660+0000 7fa0753eb700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 0x7fa05c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.661+0000 7fa0753eb700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 0x7fa05c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa06c0096a0 tx=0x7fa06c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.665+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa064014420 con 0x7fa0700f6510 2024-02-19T22:16:52.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.869+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa070108210 con 0x7fa0700f6510 2024-02-19T22:16:52.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.869+0000 7fa0627fc700 1 -- 172.21.15.97:0/182155807 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa064010230 con 0x7fa0700f6510 2024-02-19T22:16:52.871 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:52.872 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 msgr2=0x7fa05c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 0x7fa05c0323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa06c0096a0 tx=0x7fa06c006b40).stop 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 msgr2=0x7fa0701067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fa064005010 tx=0x7fa06400b040).stop 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 shutdown_connections 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa05c02ff10 0x7fa05c0323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 --2- 172.21.15.97:0/182155807 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0700f6510 0x7fa0701067e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 >> 172.21.15.97:0/182155807 conn(0x7fa0700f1c70 msgr2=0x7fa0700f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 shutdown_connections 2024-02-19T22:16:52.873 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:52.871+0000 7fa077e50700 1 -- 172.21.15.97:0/182155807 wait complete. 2024-02-19T22:16:52.874 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:53.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:51.807780+0000 mgr.smithi045.bvlamo (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:52.870616+0000 mon.smithi045 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.97:0/182155807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:54.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:54.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:55.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:53.808071+0000 mgr.smithi045.bvlamo (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- 172.21.15.97:0/714544578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 msgr2=0x7f60840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 --2- 172.21.15.97:0/714544578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f60840f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f6078009fb0 tx=0x7f6078009b30).stop 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- 172.21.15.97:0/714544578 shutdown_connections 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 --2- 172.21.15.97:0/714544578 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f60840f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- 172.21.15.97:0/714544578 >> 172.21.15.97:0/714544578 conn(0x7f60840f1c70 msgr2=0x7f60840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- 172.21.15.97:0/714544578 shutdown_connections 2024-02-19T22:16:56.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- 172.21.15.97:0/714544578 wait complete. 2024-02-19T22:16:56.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 Processor -- start 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.411+0000 7f608af92700 1 -- start start 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f608af92700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f608af92700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6084069e20 con 0x7f60840f6510 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f6088d2e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f6088d2e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50486/0 (socket says 172.21.15.97:50486) 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f6088d2e700 1 -- 172.21.15.97:0/1703312733 learned_addr learned my addr 172.21.15.97:0/1703312733 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.412+0000 7f6088d2e700 1 -- 172.21.15.97:0/1703312733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078005040 con 0x7f60840f6510 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f6088d2e700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6078005010 tx=0x7f607800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6078009320 con 0x7f60840f6510 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6084069fc0 con 0x7f60840f6510 2024-02-19T22:16:56.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6078013380 con 0x7f60840f6510 2024-02-19T22:16:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f608406a450 con 0x7f60840f6510 2024-02-19T22:16:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.413+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6078014e50 con 0x7f60840f6510 2024-02-19T22:16:56.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60780095f0 con 0x7f60840f6510 2024-02-19T22:16:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6084059100 con 0x7f60840f6510 2024-02-19T22:16:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f60817fa700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 0x7f60740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6078041010 con 0x7f60840f6510 2024-02-19T22:16:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f6083fff700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 0x7f60740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:56.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.414+0000 7f6083fff700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 0x7f60740323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f60700096a0 tx=0x7f6070006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:56.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.419+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6078014420 con 0x7f60840f6510 2024-02-19T22:16:56.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.625+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6084045ad0 con 0x7f60840f6510 2024-02-19T22:16:56.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.625+0000 7f60817fa700 1 -- 172.21.15.97:0/1703312733 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f607801d9d0 con 0x7f60840f6510 2024-02-19T22:16:56.627 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:16:56.628 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 msgr2=0x7f60740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 0x7f60740323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f60700096a0 tx=0x7f6070006b40).stop 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 msgr2=0x7f608406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f6078005010 tx=0x7f607800b040).stop 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 shutdown_connections 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f607402ff10 0x7f60740323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 --2- 172.21.15.97:0/1703312733 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f60840f6510 0x7f608406b9a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 >> 172.21.15.97:0/1703312733 conn(0x7f60840f1c70 msgr2=0x7f60840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 shutdown_connections 2024-02-19T22:16:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:16:56.627+0000 7f608af92700 1 -- 172.21.15.97:0/1703312733 wait complete. 2024-02-19T22:16:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:16:57.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:55.808404+0000 mgr.smithi045.bvlamo (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:16:56.626512+0000 mon.smithi045 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.97:0/1703312733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:58.240 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:58.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:59.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:16:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:57.808673+0000 mgr.smithi045.bvlamo (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 -- 172.21.15.97:0/3413485236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 msgr2=0x7fe10c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3413485236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c0f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe108009fb0 tx=0x7fe108009b30).stop 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 -- 172.21.15.97:0/3413485236 shutdown_connections 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3413485236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c0f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 -- 172.21.15.97:0/3413485236 >> 172.21.15.97:0/3413485236 conn(0x7fe10c0f0b70 msgr2=0x7fe10c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 -- 172.21.15.97:0/3413485236 shutdown_connections 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.118+0000 7fe113ec3700 1 -- 172.21.15.97:0/3413485236 wait complete. 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe113ec3700 1 Processor -- start 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe113ec3700 1 -- start start 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe113ec3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:00.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe113ec3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10c105bd0 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe111c5f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe111c5f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50500/0 (socket says 172.21.15.97:50500) 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.119+0000 7fe111c5f700 1 -- 172.21.15.97:0/3450299028 learned_addr learned my addr 172.21.15.97:0/3450299028 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe111c5f700 1 -- 172.21.15.97:0/3450299028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe108005040 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe111c5f700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fe1080089f0 tx=0x7fe10800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe108009320 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c105dd0 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe108013380 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10801ed60 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.120+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c106260 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.121+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe10801e420 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.121+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c045ad0 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.121+0000 7fe1027fc700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.121+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe108040ff0 con 0x7fe10c0f5410 2024-02-19T22:17:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.121+0000 7fe11145e700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:00.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.122+0000 7fe11145e700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 0x7fe0f8032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe0fc0096a0 tx=0x7fe0fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:00.138 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.126+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1080095f0 con 0x7fe10c0f5410 2024-02-19T22:17:00.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.331+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe10c107110 con 0x7fe10c0f5410 2024-02-19T22:17:00.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.332+0000 7fe1027fc700 1 -- 172.21.15.97:0/3450299028 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe108004490 con 0x7fe10c0f5410 2024-02-19T22:17:00.334 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:00.334 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:00.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 msgr2=0x7fe0f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 0x7fe0f8032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fe0fc0096a0 tx=0x7fe0fc006b40).stop 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 msgr2=0x7fe10c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fe1080089f0 tx=0x7fe10800b040).stop 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 shutdown_connections 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe0f802ff60 0x7fe0f8032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 --2- 172.21.15.97:0/3450299028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe10c0f5410 0x7fe10c1056e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.333+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 >> 172.21.15.97:0/3450299028 conn(0x7fe10c0f0b70 msgr2=0x7fe10c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.334+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 shutdown_connections 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:00.334+0000 7fe113ec3700 1 -- 172.21.15.97:0/3450299028 wait complete. 2024-02-19T22:17:00.336 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:00.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:00.332848+0000 mon.smithi045 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.97:0/3450299028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:16:59.809032+0000 mgr.smithi045.bvlamo (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:02.072 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:02.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:03.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:01.809283+0000 mgr.smithi045.bvlamo (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 -- 172.21.15.97:0/3647509266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 msgr2=0x7f51840f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 --2- 172.21.15.97:0/3647509266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f51840f8f80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f5174009fb0 tx=0x7f5174009b30).stop 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 -- 172.21.15.97:0/3647509266 shutdown_connections 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 --2- 172.21.15.97:0/3647509266 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f51840f8f80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 -- 172.21.15.97:0/3647509266 >> 172.21.15.97:0/3647509266 conn(0x7f51840f09a0 msgr2=0x7f51840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 -- 172.21.15.97:0/3647509266 shutdown_connections 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.958+0000 7f51889ac700 1 -- 172.21.15.97:0/3647509266 wait complete. 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51889ac700 1 Processor -- start 2024-02-19T22:17:03.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51889ac700 1 -- start start 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51889ac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51889ac700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5184106b00 con 0x7f5184060180 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50512/0 (socket says 172.21.15.97:50512) 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.959+0000 7f51837fe700 1 -- 172.21.15.97:0/2315063164 learned_addr learned my addr 172.21.15.97:0/2315063164 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f51837fe700 1 -- 172.21.15.97:0/2315063164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174005040 con 0x7f5184060180 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f51837fe700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f51740089f0 tx=0x7f517400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5174009320 con 0x7f5184060180 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5184106d00 con 0x7f5184060180 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5174013380 con 0x7f5184060180 2024-02-19T22:17:03.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f517401ed80 con 0x7f5184060180 2024-02-19T22:17:03.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.960+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5184107190 con 0x7f5184060180 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.961+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f517401e420 con 0x7f5184060180 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.961+0000 7f516bfff700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 0x7f516c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.961+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5184045ad0 con 0x7f5184060180 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.961+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5174040ff0 con 0x7f5184060180 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.961+0000 7f5182ffd700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 0x7f516c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:03.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.962+0000 7f5182ffd700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 0x7f516c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f51780096a0 tx=0x7f5178006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:03.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:03.966+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f517406d2b0 con 0x7f5184060180 2024-02-19T22:17:04.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.169+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5184107fe0 con 0x7f5184060180 2024-02-19T22:17:04.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.170+0000 7f516bfff700 1 -- 172.21.15.97:0/2315063164 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f517402c2d0 con 0x7f5184060180 2024-02-19T22:17:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:04.172 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:04.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 msgr2=0x7f516c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 0x7f516c032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f51780096a0 tx=0x7f5178006b40).stop 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 msgr2=0x7f5184106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f51740089f0 tx=0x7f517400b040).stop 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 shutdown_connections 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f516c02ff60 0x7f516c032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 --2- 172.21.15.97:0/2315063164 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5184060180 0x7f5184106610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 >> 172.21.15.97:0/2315063164 conn(0x7f51840f09a0 msgr2=0x7f51840f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 shutdown_connections 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:04.171+0000 7f51889ac700 1 -- 172.21.15.97:0/2315063164 wait complete. 2024-02-19T22:17:04.174 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:03.931958+0000 mon.smithi045 (mon.0) 157 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:17:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:03.933472+0000 mon.smithi045 (mon.0) 158 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:17:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:04.170869+0000 mon.smithi045 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.97:0/2315063164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:05.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:05.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:05.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:03.809630+0000 mgr.smithi045.bvlamo (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 -- 172.21.15.97:0/3458053076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 msgr2=0x7f3a900f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 --2- 172.21.15.97:0/3458053076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a900f68a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3a80009fb0 tx=0x7f3a80009b30).stop 2024-02-19T22:17:07.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 -- 172.21.15.97:0/3458053076 shutdown_connections 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 --2- 172.21.15.97:0/3458053076 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a900f68a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 -- 172.21.15.97:0/3458053076 >> 172.21.15.97:0/3458053076 conn(0x7f3a900f1c00 msgr2=0x7f3a900f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.793+0000 7f3a9865d700 1 -- 172.21.15.97:0/3458053076 shutdown_connections 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a9865d700 1 -- 172.21.15.97:0/3458053076 wait complete. 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a9865d700 1 Processor -- start 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a9865d700 1 -- start start 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a9865d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a9865d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a90106ce0 con 0x7f3a900f6480 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.794+0000 7f3a963f9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a963f9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51190/0 (socket says 172.21.15.97:51190) 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a963f9700 1 -- 172.21.15.97:0/568512886 learned_addr learned my addr 172.21.15.97:0/568512886 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a963f9700 1 -- 172.21.15.97:0/568512886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80005040 con 0x7f3a900f6480 2024-02-19T22:17:07.798 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a963f9700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3a80005010 tx=0x7f3a8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a80009320 con 0x7f3a900f6480 2024-02-19T22:17:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a90106ee0 con 0x7f3a900f6480 2024-02-19T22:17:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a80013380 con 0x7f3a900f6480 2024-02-19T22:17:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a80014e50 con 0x7f3a900f6480 2024-02-19T22:17:07.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.795+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a90107370 con 0x7f3a900f6480 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.796+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3a800263a0 con 0x7f3a900f6480 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.796+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a90045ad0 con 0x7f3a900f6480 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.796+0000 7f3a86ffd700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 0x7f3a7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.796+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3a80041140 con 0x7f3a900f6480 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.796+0000 7f3a95bf8700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 0x7f3a7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:07.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.797+0000 7f3a95bf8700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 0x7f3a7c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3a8c0096a0 tx=0x7f3a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:07.803 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:07.801+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3a8001cb10 con 0x7f3a900f6480 2024-02-19T22:17:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:05.809922+0000 mgr.smithi045.bvlamo (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:08.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.007+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a900fc090 con 0x7f3a900f6480 2024-02-19T22:17:08.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.007+0000 7f3a86ffd700 1 -- 172.21.15.97:0/568512886 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a8001cb10 con 0x7f3a900f6480 2024-02-19T22:17:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:08.009 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 msgr2=0x7f3a7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 0x7f3a7c0323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3a8c0096a0 tx=0x7f3a8c006b40).stop 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 msgr2=0x7f3a901067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3a80005010 tx=0x7f3a8000b040).stop 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 shutdown_connections 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3a7c02ff10 0x7f3a7c0323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 --2- 172.21.15.97:0/568512886 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3a900f6480 0x7f3a901067f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 >> 172.21.15.97:0/568512886 conn(0x7f3a900f1c00 msgr2=0x7f3a900f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 shutdown_connections 2024-02-19T22:17:08.011 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:08.009+0000 7f3a9865d700 1 -- 172.21.15.97:0/568512886 wait complete. 2024-02-19T22:17:08.012 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:08.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:08.008335+0000 mon.smithi045 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.97:0/568512886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:09.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:09.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:07.810174+0000 mgr.smithi045.bvlamo (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.675+0000 7f49d3adf700 1 -- 172.21.15.97:0/469113018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc0f5410 msgr2=0x7f49cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.675+0000 7f49d3adf700 1 --2- 172.21.15.97:0/469113018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc0f5410 0x7f49cc0f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f49c0009fb0 tx=0x7f49c0009b30).stop 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.675+0000 7f49d3adf700 1 -- 172.21.15.97:0/469113018 shutdown_connections 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.675+0000 7f49d3adf700 1 --2- 172.21.15.97:0/469113018 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc0f5410 0x7f49cc0f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.675+0000 7f49d3adf700 1 -- 172.21.15.97:0/469113018 >> 172.21.15.97:0/469113018 conn(0x7f49cc0f0b70 msgr2=0x7f49cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 -- 172.21.15.97:0/469113018 shutdown_connections 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 -- 172.21.15.97:0/469113018 wait complete. 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 Processor -- start 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 -- start start 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d3adf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49cc1060f0 con 0x7f49cc1057e0 2024-02-19T22:17:11.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.676+0000 7f49d187b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d187b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51218/0 (socket says 172.21.15.97:51218) 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d187b700 1 -- 172.21.15.97:0/620419277 learned_addr learned my addr 172.21.15.97:0/620419277 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d187b700 1 -- 172.21.15.97:0/620419277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0005040 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d187b700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f49cc0f6510 tx=0x7f49c0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c0013380 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49cc1062f0 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49c0013790 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49c001ee70 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.677+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49cc106e40 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.678+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49c0025060 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.678+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49cc059100 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.678+0000 7f49be7fc700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 0x7f49b8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.678+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f49c0040db0 con 0x7f49cc1057e0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.678+0000 7f49d107a700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 0x7f49b8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.679+0000 7f49d107a700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 0x7f49b8032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f49c80096a0 tx=0x7f49c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.683+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f49c001c520 con 0x7f49cc1057e0 2024-02-19T22:17:11.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.885+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49cc045ad0 con 0x7f49cc1057e0 2024-02-19T22:17:11.887 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.886+0000 7f49be7fc700 1 -- 172.21.15.97:0/620419277 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49c002d2d0 con 0x7f49cc1057e0 2024-02-19T22:17:11.888 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:11.888 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 msgr2=0x7f49b8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 0x7f49b8032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f49c80096a0 tx=0x7f49c8006b40).stop 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 msgr2=0x7f49cc105c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f49cc0f6510 tx=0x7f49c0009b30).stop 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 shutdown_connections 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f49b802ffb0 0x7f49b8032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 --2- 172.21.15.97:0/620419277 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f49cc1057e0 0x7f49cc105c00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 >> 172.21.15.97:0/620419277 conn(0x7f49cc0f0b70 msgr2=0x7f49cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 shutdown_connections 2024-02-19T22:17:11.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:11.888+0000 7f49d3adf700 1 -- 172.21.15.97:0/620419277 wait complete. 2024-02-19T22:17:11.891 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:09.810498+0000 mgr.smithi045.bvlamo (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:12.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:12 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:11.887016+0000 mon.smithi045 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.97:0/620419277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:13.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:13.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:11.810870+0000 mgr.smithi045.bvlamo (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:15.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 -- 172.21.15.97:0/648142967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 msgr2=0x7fb8240f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/648142967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb8240f8f80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb820009fb0 tx=0x7fb820009b30).stop 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 -- 172.21.15.97:0/648142967 shutdown_connections 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/648142967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb8240f8f80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 -- 172.21.15.97:0/648142967 >> 172.21.15.97:0/648142967 conn(0x7fb8240f09a0 msgr2=0x7fb8240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 -- 172.21.15.97:0/648142967 shutdown_connections 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.540+0000 7fb82a5e3700 1 -- 172.21.15.97:0/648142967 wait complete. 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb82a5e3700 1 Processor -- start 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb82a5e3700 1 -- start start 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb82a5e3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb82a5e3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb824106b00 con 0x7fb824060180 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb8295e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb8295e1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51232/0 (socket says 172.21.15.97:51232) 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.541+0000 7fb8295e1700 1 -- 172.21.15.97:0/1257589192 learned_addr learned my addr 172.21.15.97:0/1257589192 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:15.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb8295e1700 1 -- 172.21.15.97:0/1257589192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb820005040 con 0x7fb824060180 2024-02-19T22:17:15.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb8295e1700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb820005010 tx=0x7fb82000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb820009320 con 0x7fb824060180 2024-02-19T22:17:15.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb820013380 con 0x7fb824060180 2024-02-19T22:17:15.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824106d00 con 0x7fb824060180 2024-02-19T22:17:15.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb820014e50 con 0x7fb824060180 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.542+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb824107190 con 0x7fb824060180 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.543+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb824045ad0 con 0x7fb824060180 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.543+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8200095f0 con 0x7fb824060180 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.543+0000 7fb819ffb700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 0x7fb8100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.543+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb820040ef0 con 0x7fb824060180 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.543+0000 7fb828de0700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 0x7fb8100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.544+0000 7fb828de0700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 0x7fb8100323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb8140096a0 tx=0x7fb814006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.549 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.548+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb820014420 con 0x7fb824060180 2024-02-19T22:17:15.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.753+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb824107fe0 con 0x7fb824060180 2024-02-19T22:17:15.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.754+0000 7fb819ffb700 1 -- 172.21.15.97:0/1257589192 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb820010230 con 0x7fb824060180 2024-02-19T22:17:15.755 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:15.756 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 msgr2=0x7fb8100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 0x7fb8100323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb8140096a0 tx=0x7fb814006b40).stop 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 msgr2=0x7fb824106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb820005010 tx=0x7fb82000b040).stop 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 shutdown_connections 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb81002ff10 0x7fb8100323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 --2- 172.21.15.97:0/1257589192 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb824060180 0x7fb824106610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 >> 172.21.15.97:0/1257589192 conn(0x7fb8240f09a0 msgr2=0x7fb8240f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 shutdown_connections 2024-02-19T22:17:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:15.755+0000 7fb82a5e3700 1 -- 172.21.15.97:0/1257589192 wait complete. 2024-02-19T22:17:15.758 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:13.811193+0000 mgr.smithi045.bvlamo (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:16.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:15.754749+0000 mon.smithi045 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.97:0/1257589192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:17.395 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:17.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:17.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:15.811531+0000 mgr.smithi045.bvlamo (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:18.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:17.811833+0000 mgr.smithi045.bvlamo (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:19.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 -- 172.21.15.97:0/1651442117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 msgr2=0x7efcb00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 --2- 172.21.15.97:0/1651442117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb00f4b60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7efca0009fb0 tx=0x7efca0009b30).stop 2024-02-19T22:17:19.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 -- 172.21.15.97:0/1651442117 shutdown_connections 2024-02-19T22:17:19.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 --2- 172.21.15.97:0/1651442117 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb00f4b60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 -- 172.21.15.97:0/1651442117 >> 172.21.15.97:0/1651442117 conn(0x7efcb00f0260 msgr2=0x7efcb00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 -- 172.21.15.97:0/1651442117 shutdown_connections 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.409+0000 7efcb87b9700 1 -- 172.21.15.97:0/1651442117 wait complete. 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb87b9700 1 Processor -- start 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb87b9700 1 -- start start 2024-02-19T22:17:19.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb87b9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb87b9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0106ca0 con 0x7efcb00f4740 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb6555700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb6555700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54872/0 (socket says 172.21.15.97:54872) 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb6555700 1 -- 172.21.15.97:0/2346173056 learned_addr learned my addr 172.21.15.97:0/2346173056 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.410+0000 7efcb6555700 1 -- 172.21.15.97:0/2346173056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca0005040 con 0x7efcb00f4740 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efcb6555700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efca0005010 tx=0x7efca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:19.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efca0009320 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb0106ea0 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efca0013380 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efca0014e50 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb0107330 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.411+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb0045ad0 con 0x7efcb00f4740 2024-02-19T22:17:19.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.412+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efca0014420 con 0x7efcb00f4740 2024-02-19T22:17:19.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.412+0000 7efca6ffd700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 0x7efc9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:19.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.412+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efca0040e30 con 0x7efcb00f4740 2024-02-19T22:17:19.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.412+0000 7efcb5d54700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 0x7efc9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:19.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.412+0000 7efcb5d54700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 0x7efc9c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7efcac0096a0 tx=0x7efcac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:19.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.417+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efca00095f0 con 0x7efcb00f4740 2024-02-19T22:17:19.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.625+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efcb01081e0 con 0x7efcb00f4740 2024-02-19T22:17:19.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.626+0000 7efca6ffd700 1 -- 172.21.15.97:0/2346173056 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efca001db70 con 0x7efcb00f4740 2024-02-19T22:17:19.628 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:19.628 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:19.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 msgr2=0x7efc9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 0x7efc9c0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7efcac0096a0 tx=0x7efcac006b40).stop 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 msgr2=0x7efcb01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efca0005010 tx=0x7efca000b040).stop 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 shutdown_connections 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efc9c02ff10 0x7efc9c0323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 --2- 172.21.15.97:0/2346173056 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efcb00f4740 0x7efcb01067b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 >> 172.21.15.97:0/2346173056 conn(0x7efcb00f0260 msgr2=0x7efcb00f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 shutdown_connections 2024-02-19T22:17:19.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:19.628+0000 7efcb87b9700 1 -- 172.21.15.97:0/2346173056 wait complete. 2024-02-19T22:17:19.631 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:20.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:19.627066+0000 mon.smithi045 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.97:0/2346173056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:21.140 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:19.812060+0000 mgr.smithi045.bvlamo (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:21.340 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:21.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:23.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 -- 172.21.15.97:0/795844722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 msgr2=0x7fa8cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:23.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 --2- 172.21.15.97:0/795844722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc0f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fa8bc009fb0 tx=0x7fa8bc009b30).stop 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 -- 172.21.15.97:0/795844722 shutdown_connections 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 --2- 172.21.15.97:0/795844722 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc0f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 -- 172.21.15.97:0/795844722 >> 172.21.15.97:0/795844722 conn(0x7fa8cc0f0b70 msgr2=0x7fa8cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 -- 172.21.15.97:0/795844722 shutdown_connections 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.168+0000 7fa8d085a700 1 -- 172.21.15.97:0/795844722 wait complete. 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8d085a700 1 Processor -- start 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8d085a700 1 -- start start 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8d085a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8d085a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc105bd0 con 0x7fa8cc0f5410 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8ca59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8ca59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54886/0 (socket says 172.21.15.97:54886) 2024-02-19T22:17:23.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8ca59c700 1 -- 172.21.15.97:0/156371187 learned_addr learned my addr 172.21.15.97:0/156371187 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8ca59c700 1 -- 172.21.15.97:0/156371187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8bc005040 con 0x7fa8cc0f5410 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.169+0000 7fa8ca59c700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa8bc005010 tx=0x7fa8bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.170+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc009320 con 0x7fa8cc0f5410 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.170+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc105dd0 con 0x7fa8cc0f5410 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.170+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc106260 con 0x7fa8cc0f5410 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.170+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8bc013380 con 0x7fa8cc0f5410 2024-02-19T22:17:23.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.170+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8bc014df0 con 0x7fa8cc0f5410 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8bc014420 con 0x7fa8cc0f5410 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc045ad0 con 0x7fa8cc0f5410 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8baffd700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 0x7fa8b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8bc0410e0 con 0x7fa8cc0f5410 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8c9d9b700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 0x7fa8b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:23.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.171+0000 7fa8c9d9b700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 0x7fa8b40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa8c00096a0 tx=0x7fa8c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:23.177 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.176+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8bc00fd10 con 0x7fa8cc0f5410 2024-02-19T22:17:23.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.379+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8cc107110 con 0x7fa8cc0f5410 2024-02-19T22:17:23.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.380+0000 7fa8baffd700 1 -- 172.21.15.97:0/156371187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8bc00fb30 con 0x7fa8cc0f5410 2024-02-19T22:17:23.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:23.382 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 msgr2=0x7fa8b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 0x7fa8b40323d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7fa8c00096a0 tx=0x7fa8c0006b40).stop 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 msgr2=0x7fa8cc1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fa8bc005010 tx=0x7fa8bc00b040).stop 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 shutdown_connections 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8b402ff10 0x7fa8b40323d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 --2- 172.21.15.97:0/156371187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa8cc0f5410 0x7fa8cc1056e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:23.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 >> 172.21.15.97:0/156371187 conn(0x7fa8cc0f0b70 msgr2=0x7fa8cc0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:23.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 shutdown_connections 2024-02-19T22:17:23.385 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:23.382+0000 7fa8d085a700 1 -- 172.21.15.97:0/156371187 wait complete. 2024-02-19T22:17:23.385 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:21.812371+0000 mgr.smithi045.bvlamo (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:23.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:23.381228+0000 mon.smithi045 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.97:0/156371187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:25.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:25.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:25.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:23.812726+0000 mgr.smithi045.bvlamo (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:27.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 -- 172.21.15.97:0/1029817150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 msgr2=0x7f768c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 --2- 172.21.15.97:0/1029817150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c0f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f7680009fb0 tx=0x7f7680009b30).stop 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 -- 172.21.15.97:0/1029817150 shutdown_connections 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 --2- 172.21.15.97:0/1029817150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c0f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 -- 172.21.15.97:0/1029817150 >> 172.21.15.97:0/1029817150 conn(0x7f768c0f1c70 msgr2=0x7f768c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 -- 172.21.15.97:0/1029817150 shutdown_connections 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.076+0000 7f7691474700 1 -- 172.21.15.97:0/1029817150 wait complete. 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f7691474700 1 Processor -- start 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f7691474700 1 -- start start 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f7691474700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:27.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f7691474700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c069e20 con 0x7f768c0f6510 2024-02-19T22:17:27.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f768affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:27.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f768affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48358/0 (socket says 172.21.15.97:48358) 2024-02-19T22:17:27.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.077+0000 7f768affd700 1 -- 172.21.15.97:0/3971226307 learned_addr learned my addr 172.21.15.97:0/3971226307 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:27.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f768affd700 1 -- 172.21.15.97:0/3971226307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7680005040 con 0x7f768c0f6510 2024-02-19T22:17:27.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f768affd700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7680005010 tx=0x7f768000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7680009320 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c069fc0 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7680013380 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7680014e50 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.078+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c06a450 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.079+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76800263a0 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.079+0000 7f76737fe700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 0x7f76740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.079+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7680040fa0 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.079+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768c059100 con 0x7f768c0f6510 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.079+0000 7f768a7fc700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 0x7f76740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.080+0000 7f768a7fc700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 0x7f76740323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f767c0096a0 tx=0x7f767c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:27.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.084+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7680012920 con 0x7f768c0f6510 2024-02-19T22:17:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.290+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f768c06aab0 con 0x7f768c0f6510 2024-02-19T22:17:27.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.290+0000 7f76737fe700 1 -- 172.21.15.97:0/3971226307 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f768002b020 con 0x7f768c0f6510 2024-02-19T22:17:27.292 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:27.293 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 msgr2=0x7f76740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 0x7f76740323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f767c0096a0 tx=0x7f767c006b40).stop 2024-02-19T22:17:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 msgr2=0x7f768c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f7680005010 tx=0x7f768000b040).stop 2024-02-19T22:17:27.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 shutdown_connections 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f767402ff10 0x7f76740323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 --2- 172.21.15.97:0/3971226307 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f768c0f6510 0x7f768c06b9a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 >> 172.21.15.97:0/3971226307 conn(0x7f768c0f1c70 msgr2=0x7f768c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 shutdown_connections 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:27.292+0000 7f7691474700 1 -- 172.21.15.97:0/3971226307 wait complete. 2024-02-19T22:17:27.295 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:25.813076+0000 mgr.smithi045.bvlamo (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:27.291612+0000 mon.smithi045 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.97:0/3971226307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:28.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:28.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:27.813352+0000 mgr.smithi045.bvlamo (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- 172.21.15.97:0/1233270358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 msgr2=0x7fd5c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 --2- 172.21.15.97:0/1233270358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c40f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fd5b8009fb0 tx=0x7fd5b8009b30).stop 2024-02-19T22:17:30.953 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- 172.21.15.97:0/1233270358 shutdown_connections 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 --2- 172.21.15.97:0/1233270358 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c40f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- 172.21.15.97:0/1233270358 >> 172.21.15.97:0/1233270358 conn(0x7fd5c40f1c70 msgr2=0x7fd5c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- 172.21.15.97:0/1233270358 shutdown_connections 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- 172.21.15.97:0/1233270358 wait complete. 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 Processor -- start 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.951+0000 7fd5caf13700 1 -- start start 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5caf13700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.954 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5caf13700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4069e20 con 0x7fd5c40f6510 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5c8caf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5c8caf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48372/0 (socket says 172.21.15.97:48372) 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5c8caf700 1 -- 172.21.15.97:0/3648495974 learned_addr learned my addr 172.21.15.97:0/3648495974 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5c8caf700 1 -- 172.21.15.97:0/3648495974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8005040 con 0x7fd5c40f6510 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.952+0000 7fd5c8caf700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd5b8005010 tx=0x7fd5b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b8009320 con 0x7fd5c40f6510 2024-02-19T22:17:30.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c4069fc0 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5b8013380 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5b8014e50 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c406a450 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5b80095f0 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.953+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c4059100 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.954+0000 7fd5c17fa700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.954+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5b8040f70 con 0x7fd5c40f6510 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.954+0000 7fd5c3fff700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.954+0000 7fd5c3fff700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 0x7fd5b40323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5b00096a0 tx=0x7fd5b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:30.959+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5b8014420 con 0x7fd5c40f6510 2024-02-19T22:17:31.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.163+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5c4045ad0 con 0x7fd5c40f6510 2024-02-19T22:17:31.165 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.164+0000 7fd5c17fa700 1 -- 172.21.15.97:0/3648495974 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd5b8010230 con 0x7fd5c40f6510 2024-02-19T22:17:31.166 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:31.166 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:31.167 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 msgr2=0x7fd5b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 0x7fd5b40323d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fd5b00096a0 tx=0x7fd5b0006b40).stop 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 msgr2=0x7fd5c406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fd5b8005010 tx=0x7fd5b800b040).stop 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 shutdown_connections 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd5b402ff10 0x7fd5b40323d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 --2- 172.21.15.97:0/3648495974 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd5c40f6510 0x7fd5c406b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 >> 172.21.15.97:0/3648495974 conn(0x7fd5c40f1c70 msgr2=0x7fd5c40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 shutdown_connections 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:31.166+0000 7fd5caf13700 1 -- 172.21.15.97:0/3648495974 wait complete. 2024-02-19T22:17:31.168 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:31.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:29.813604+0000 mgr.smithi045.bvlamo (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:31.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:31.164914+0000 mon.smithi045 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.97:0/3648495974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:32.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:32.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:33.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:31.813960+0000 mgr.smithi045.bvlamo (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:34.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 -- 172.21.15.97:0/3050911238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 msgr2=0x7f8f200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3050911238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f200f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f8f10009fb0 tx=0x7f8f10009b30).stop 2024-02-19T22:17:34.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 -- 172.21.15.97:0/3050911238 shutdown_connections 2024-02-19T22:17:34.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3050911238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f200f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 -- 172.21.15.97:0/3050911238 >> 172.21.15.97:0/3050911238 conn(0x7f8f200f0b70 msgr2=0x7f8f200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:34.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 -- 172.21.15.97:0/3050911238 shutdown_connections 2024-02-19T22:17:34.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.654+0000 7f8f253f6700 1 -- 172.21.15.97:0/3050911238 wait complete. 2024-02-19T22:17:34.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.655+0000 7f8f253f6700 1 Processor -- start 2024-02-19T22:17:34.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.655+0000 7f8f253f6700 1 -- start start 2024-02-19T22:17:34.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.655+0000 7f8f253f6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:34.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.655+0000 7f8f253f6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f20106cd0 con 0x7f8f200f5410 2024-02-19T22:17:34.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f1effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48388/0 (socket says 172.21.15.97:48388) 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f1effd700 1 -- 172.21.15.97:0/3529084150 learned_addr learned my addr 172.21.15.97:0/3529084150 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f1effd700 1 -- 172.21.15.97:0/3529084150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f10005040 con 0x7f8f200f5410 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f1effd700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8f10005010 tx=0x7f8f1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f10009320 con 0x7f8f200f5410 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f20106ed0 con 0x7f8f200f5410 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f10013380 con 0x7f8f200f5410 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f10014e50 con 0x7f8f200f5410 2024-02-19T22:17:34.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.656+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f20107360 con 0x7f8f200f5410 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.657+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f100095f0 con 0x7f8f200f5410 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.657+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f20045ad0 con 0x7f8f200f5410 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.657+0000 7f8f077fe700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.657+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f10040e70 con 0x7f8f200f5410 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.657+0000 7f8f1e7fc700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:34.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.658+0000 7f8f1e7fc700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 0x7f8f080323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:34.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.662+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f10014420 con 0x7f8f200f5410 2024-02-19T22:17:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.866+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f201081b0 con 0x7f8f200f5410 2024-02-19T22:17:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.867+0000 7f8f077fe700 1 -- 172.21.15.97:0/3529084150 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8f10010230 con 0x7f8f200f5410 2024-02-19T22:17:34.868 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:34.868 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 msgr2=0x7f8f080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 0x7f8f080323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f8f140096a0 tx=0x7f8f14006b40).stop 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 msgr2=0x7f8f201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f8f10005010 tx=0x7f8f1000b040).stop 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 shutdown_connections 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8f0802ff10 0x7f8f080323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 --2- 172.21.15.97:0/3529084150 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8f200f5410 0x7f8f201067e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 >> 172.21.15.97:0/3529084150 conn(0x7f8f200f0b70 msgr2=0x7f8f200f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 shutdown_connections 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:34.868+0000 7f8f253f6700 1 -- 172.21.15.97:0/3529084150 wait complete. 2024-02-19T22:17:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:35.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:33.814249+0000 mgr.smithi045.bvlamo (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:35.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:34.867741+0000 mon.smithi045 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.97:0/3529084150' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:36.485 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:36.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:37.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:35.814547+0000 mgr.smithi045.bvlamo (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:38.294 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1901883899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 msgr2=0x7efc140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1901883899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc140f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7efc04009fb0 tx=0x7efc04009b30).stop 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1901883899 shutdown_connections 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1901883899 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc140f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1901883899 >> 172.21.15.97:0/1901883899 conn(0x7efc140f0b70 msgr2=0x7efc140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1901883899 shutdown_connections 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.290+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1901883899 wait complete. 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc1a8a1700 1 Processor -- start 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc1a8a1700 1 -- start start 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc1a8a1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:38.295 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc1a8a1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14106cc0 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc13fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33934/0 (socket says 172.21.15.97:33934) 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc13fff700 1 -- 172.21.15.97:0/1654057165 learned_addr learned my addr 172.21.15.97:0/1654057165 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.291+0000 7efc13fff700 1 -- 172.21.15.97:0/1654057165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc04005040 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc13fff700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7efc04005010 tx=0x7efc0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc04009320 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc14106ec0 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc04013380 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc04014e50 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.292+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc14107350 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc040263a0 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc14045ad0 con 0x7efc140f5410 2024-02-19T22:17:38.296 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc10ff9700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:38.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc04041240 con 0x7efc140f5410 2024-02-19T22:17:38.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc137fe700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:38.297 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.293+0000 7efc137fe700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 0x7efbfc0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7efc080096a0 tx=0x7efc08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:38.299 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.297+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc040126f0 con 0x7efc140f5410 2024-02-19T22:17:38.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.502+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc141081a0 con 0x7efc140f5410 2024-02-19T22:17:38.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.502+0000 7efc10ff9700 1 -- 172.21.15.97:0/1654057165 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc04012330 con 0x7efc140f5410 2024-02-19T22:17:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:38.505 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 msgr2=0x7efbfc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 0x7efbfc0323d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7efc080096a0 tx=0x7efc08006b40).stop 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 msgr2=0x7efc141067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7efc04005010 tx=0x7efc0400b040).stop 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 shutdown_connections 2024-02-19T22:17:38.506 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7efbfc02ff10 0x7efbfc0323d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 --2- 172.21.15.97:0/1654057165 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7efc140f5410 0x7efc141067d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:38.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 >> 172.21.15.97:0/1654057165 conn(0x7efc140f0b70 msgr2=0x7efc1405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:38.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 shutdown_connections 2024-02-19T22:17:38.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:38.504+0000 7efc1a8a1700 1 -- 172.21.15.97:0/1654057165 wait complete. 2024-02-19T22:17:38.507 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:38.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:38.503459+0000 mon.smithi045 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.97:0/1654057165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:39.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:37.814797+0000 mgr.smithi045.bvlamo (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:40.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:40.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:41.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:39.815086+0000 mgr.smithi045.bvlamo (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:42.149 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 -- 172.21.15.97:0/508946063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 msgr2=0x7f4ba80f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/508946063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba80f66e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4b98009fb0 tx=0x7f4b98009b30).stop 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 -- 172.21.15.97:0/508946063 shutdown_connections 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/508946063 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba80f66e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 -- 172.21.15.97:0/508946063 >> 172.21.15.97:0/508946063 conn(0x7f4ba80f1ac0 msgr2=0x7f4ba80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 -- 172.21.15.97:0/508946063 shutdown_connections 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.146+0000 7f4bac9fd700 1 -- 172.21.15.97:0/508946063 wait complete. 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4bac9fd700 1 Processor -- start 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4bac9fd700 1 -- start start 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4bac9fd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4bac9fd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8106ad0 con 0x7f4ba80f62c0 2024-02-19T22:17:42.150 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4ba77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4ba77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33956/0 (socket says 172.21.15.97:33956) 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.147+0000 7f4ba77fe700 1 -- 172.21.15.97:0/3258953564 learned_addr learned my addr 172.21.15.97:0/3258953564 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4ba77fe700 1 -- 172.21.15.97:0/3258953564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98005040 con 0x7f4ba80f62c0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4ba77fe700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4b980089f0 tx=0x7f4b9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98009320 con 0x7f4ba80f62c0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8106cd0 con 0x7f4ba80f62c0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b98013380 con 0x7f4ba80f62c0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b98014e20 con 0x7f4ba80f62c0 2024-02-19T22:17:42.151 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.148+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba8107160 con 0x7f4ba80f62c0 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.149+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b98014420 con 0x7f4ba80f62c0 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.149+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8045ad0 con 0x7f4ba80f62c0 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.149+0000 7f4b8ffff700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 0x7f4b900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.149+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b98040ea0 con 0x7f4ba80f62c0 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.149+0000 7f4ba6ffd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 0x7f4b900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.150+0000 7f4ba6ffd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 0x7f4b900323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4b9c0096a0 tx=0x7f4b9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:42.155 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.154+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b9801d980 con 0x7f4ba80f62c0 2024-02-19T22:17:42.362 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.360+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ba8108010 con 0x7f4ba80f62c0 2024-02-19T22:17:42.363 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.362+0000 7f4b8ffff700 1 -- 172.21.15.97:0/3258953564 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b98010230 con 0x7f4ba80f62c0 2024-02-19T22:17:42.363 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:42.363 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:42.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 msgr2=0x7f4b900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 0x7f4b900323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f4b9c0096a0 tx=0x7f4b9c006b40).stop 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 msgr2=0x7f4ba81065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f4b980089f0 tx=0x7f4b9800b040).stop 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 shutdown_connections 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b9002ff10 0x7f4b900323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 --2- 172.21.15.97:0/3258953564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4ba80f62c0 0x7f4ba81065e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 >> 172.21.15.97:0/3258953564 conn(0x7f4ba80f1ac0 msgr2=0x7f4ba80f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 shutdown_connections 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:42.363+0000 7f4bac9fd700 1 -- 172.21.15.97:0/3258953564 wait complete. 2024-02-19T22:17:42.365 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:42.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:42.362169+0000 mon.smithi045 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.97:0/3258953564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:43.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:41.815404+0000 mgr.smithi045.bvlamo (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:44.065 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:44.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:43.815753+0000 mgr.smithi045.bvlamo (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 -- 172.21.15.97:0/2330099166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 msgr2=0x7f93b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/2330099166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b40f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f93a4009fb0 tx=0x7f93a4009b30).stop 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 -- 172.21.15.97:0/2330099166 shutdown_connections 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/2330099166 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b40f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 -- 172.21.15.97:0/2330099166 >> 172.21.15.97:0/2330099166 conn(0x7f93b40f0b70 msgr2=0x7f93b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 -- 172.21.15.97:0/2330099166 shutdown_connections 2024-02-19T22:17:46.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 -- 172.21.15.97:0/2330099166 wait complete. 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.033+0000 7f93bb5bc700 1 Processor -- start 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93bb5bc700 1 -- start start 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93bb5bc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93bb5bc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4105bd0 con 0x7f93b40f5410 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93b9358700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93b9358700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:33978/0 (socket says 172.21.15.97:33978) 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93b9358700 1 -- 172.21.15.97:0/3944602844 learned_addr learned my addr 172.21.15.97:0/3944602844 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.034+0000 7f93b9358700 1 -- 172.21.15.97:0/3944602844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4005040 con 0x7f93b40f5410 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93b9358700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f93a4005010 tx=0x7f93a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a4009320 con 0x7f93b40f5410 2024-02-19T22:17:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b4105dd0 con 0x7f93b40f5410 2024-02-19T22:17:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93a4013380 con 0x7f93b40f5410 2024-02-19T22:17:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a4014e50 con 0x7f93b40f5410 2024-02-19T22:17:46.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.035+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b4106260 con 0x7f93b40f5410 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.036+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93a40095f0 con 0x7f93b40f5410 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.036+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b4045ad0 con 0x7f93b40f5410 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.036+0000 7f93a9ffb700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 0x7f93a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.036+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93a4040e70 con 0x7f93b40f5410 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.036+0000 7f93b8b57700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 0x7f93a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:46.040 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.037+0000 7f93b8b57700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 0x7f93a00323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f93b00096a0 tx=0x7f93b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:46.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.041+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93a4014420 con 0x7f93b40f5410 2024-02-19T22:17:46.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.249+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93b4107110 con 0x7f93b40f5410 2024-02-19T22:17:46.251 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.250+0000 7f93a9ffb700 1 -- 172.21.15.97:0/3944602844 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f93a4010230 con 0x7f93b40f5410 2024-02-19T22:17:46.252 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:46.252 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.251+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 msgr2=0x7f93a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 0x7f93a00323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f93b00096a0 tx=0x7f93b0006b40).stop 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 msgr2=0x7f93b41056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f93a4005010 tx=0x7f93a400b040).stop 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 shutdown_connections 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f93a002ff10 0x7f93a00323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 --2- 172.21.15.97:0/3944602844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f93b40f5410 0x7f93b41056e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 >> 172.21.15.97:0/3944602844 conn(0x7f93b40f0b70 msgr2=0x7f93b40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 shutdown_connections 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:46.252+0000 7f93bb5bc700 1 -- 172.21.15.97:0/3944602844 wait complete. 2024-02-19T22:17:46.254 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:46.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:46.250874+0000 mon.smithi045 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.97:0/3944602844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:47.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:45.816134+0000 mgr.smithi045.bvlamo (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:47.943 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:47.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:49.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 -- 172.21.15.97:0/1655239224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 msgr2=0x7f1ad80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 --2- 172.21.15.97:0/1655239224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad80f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f1ac8009fb0 tx=0x7f1ac8009b30).stop 2024-02-19T22:17:49.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 -- 172.21.15.97:0/1655239224 shutdown_connections 2024-02-19T22:17:49.796 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 --2- 172.21.15.97:0/1655239224 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad80f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 -- 172.21.15.97:0/1655239224 >> 172.21.15.97:0/1655239224 conn(0x7f1ad80f1c70 msgr2=0x7f1ad80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:49.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 -- 172.21.15.97:0/1655239224 shutdown_connections 2024-02-19T22:17:49.797 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.794+0000 7f1ae0688700 1 -- 172.21.15.97:0/1655239224 wait complete. 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ae0688700 1 Processor -- start 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ae0688700 1 -- start start 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ae0688700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ae0688700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad8106cd0 con 0x7f1ad80f6510 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ade424700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ade424700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54974/0 (socket says 172.21.15.97:54974) 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.795+0000 7f1ade424700 1 -- 172.21.15.97:0/772941377 learned_addr learned my addr 172.21.15.97:0/772941377 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:49.799 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1ade424700 1 -- 172.21.15.97:0/772941377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac8005040 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1ade424700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1ac80089f0 tx=0x7f1ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac8009320 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad8106ed0 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ac8013380 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ac801ed80 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.796+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad8107360 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.797+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ac8014420 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.797+0000 7f1aceffd700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 0x7f1ac4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.797+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ac80411c0 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.797+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad8045ad0 con 0x7f1ad80f6510 2024-02-19T22:17:49.800 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.797+0000 7f1addc23700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 0x7f1ac4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.801 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.798+0000 7f1addc23700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 0x7f1ac4032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1ad40096a0 tx=0x7f1ad4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.804 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:49.802+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ac801c910 con 0x7f1ad80f6510 2024-02-19T22:17:49.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:47.816517+0000 mgr.smithi045.bvlamo (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.005+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ad80fc190 con 0x7f1ad80f6510 2024-02-19T22:17:50.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.005+0000 7f1aceffd700 1 -- 172.21.15.97:0/772941377 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1ac801c320 con 0x7f1ad80f6510 2024-02-19T22:17:50.007 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:50.007 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:50.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 msgr2=0x7f1ac4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:50.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 0x7f1ac4032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f1ad40096a0 tx=0x7f1ad4006b40).stop 2024-02-19T22:17:50.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 msgr2=0x7f1ad81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:50.009 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1ac80089f0 tx=0x7f1ac800b040).stop 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 shutdown_connections 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f1ac402ff60 0x7f1ac4032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 --2- 172.21.15.97:0/772941377 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f1ad80f6510 0x7f1ad81067e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 >> 172.21.15.97:0/772941377 conn(0x7f1ad80f1c70 msgr2=0x7f1ad80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 shutdown_connections 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:50.007+0000 7f1ae0688700 1 -- 172.21.15.97:0/772941377 wait complete. 2024-02-19T22:17:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:50.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:50.006518+0000 mon.smithi045 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.97:0/772941377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:51.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:51.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:51.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:49.816906+0000 mgr.smithi045.bvlamo (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:53.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.718+0000 7ff297614700 1 -- 172.21.15.97:0/3161836324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 msgr2=0x7ff2900f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.718+0000 7ff297614700 1 --2- 172.21.15.97:0/3161836324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2900f6930 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7ff284009fb0 tx=0x7ff284009b30).stop 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.718+0000 7ff297614700 1 -- 172.21.15.97:0/3161836324 shutdown_connections 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.718+0000 7ff297614700 1 --2- 172.21.15.97:0/3161836324 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2900f6930 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.718+0000 7ff297614700 1 -- 172.21.15.97:0/3161836324 >> 172.21.15.97:0/3161836324 conn(0x7ff2900f1c70 msgr2=0x7ff2900f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 -- 172.21.15.97:0/3161836324 shutdown_connections 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 -- 172.21.15.97:0/3161836324 wait complete. 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 Processor -- start 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 -- start start 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff297614700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff290106cd0 con 0x7ff2900f6510 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff2953b0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff2953b0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54988/0 (socket says 172.21.15.97:54988) 2024-02-19T22:17:53.723 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.719+0000 7ff2953b0700 1 -- 172.21.15.97:0/26458996 learned_addr learned my addr 172.21.15.97:0/26458996 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff2953b0700 1 -- 172.21.15.97:0/26458996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284005040 con 0x7ff2900f6510 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff2953b0700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff284005010 tx=0x7ff28400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff284009320 con 0x7ff2900f6510 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff290106ed0 con 0x7ff2900f6510 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff284013380 con 0x7ff2900f6510 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff284014e50 con 0x7ff2900f6510 2024-02-19T22:17:53.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.720+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff290107360 con 0x7ff2900f6510 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.721+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2840095f0 con 0x7ff2900f6510 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.721+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff290045ad0 con 0x7ff2900f6510 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.721+0000 7ff281ffb700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 0x7ff27c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.721+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff284040ed0 con 0x7ff2900f6510 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.722+0000 7ff294baf700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 0x7ff27c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:53.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.722+0000 7ff294baf700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 0x7ff27c0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff28c0096a0 tx=0x7ff28c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:53.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.726+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff284014420 con 0x7ff2900f6510 2024-02-19T22:17:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:51.817295+0000 mgr.smithi045.bvlamo (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:53.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.934+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff290108210 con 0x7ff2900f6510 2024-02-19T22:17:53.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.935+0000 7ff281ffb700 1 -- 172.21.15.97:0/26458996 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff284010230 con 0x7ff2900f6510 2024-02-19T22:17:53.937 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:53.938 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 msgr2=0x7ff27c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 0x7ff27c0323d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7ff28c0096a0 tx=0x7ff28c006b40).stop 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 msgr2=0x7ff2901067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7ff284005010 tx=0x7ff28400b040).stop 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 shutdown_connections 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff27c02ff10 0x7ff27c0323d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 --2- 172.21.15.97:0/26458996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff2900f6510 0x7ff2901067e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 >> 172.21.15.97:0/26458996 conn(0x7ff2900f1c70 msgr2=0x7ff2900f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 shutdown_connections 2024-02-19T22:17:53.939 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:53.936+0000 7ff297614700 1 -- 172.21.15.97:0/26458996 wait complete. 2024-02-19T22:17:53.940 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:54.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:53.935864+0000 mon.smithi045 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.97:0/26458996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:55.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:55.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:55.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:53.817670+0000 mgr.smithi045.bvlamo (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:57.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 -- 172.21.15.97:0/617508213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 msgr2=0x7fa7000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 --2- 172.21.15.97:0/617508213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7000f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fa6f0009fb0 tx=0x7fa6f0009b30).stop 2024-02-19T22:17:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 -- 172.21.15.97:0/617508213 shutdown_connections 2024-02-19T22:17:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 --2- 172.21.15.97:0/617508213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7000f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 -- 172.21.15.97:0/617508213 >> 172.21.15.97:0/617508213 conn(0x7fa7000f0b70 msgr2=0x7fa7000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:57.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 -- 172.21.15.97:0/617508213 shutdown_connections 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.436+0000 7fa7083cb700 1 -- 172.21.15.97:0/617508213 wait complete. 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa7083cb700 1 Processor -- start 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa7083cb700 1 -- start start 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa7083cb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa7083cb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa700105bd0 con 0x7fa7000f5410 2024-02-19T22:17:57.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa706167700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa706167700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53904/0 (socket says 172.21.15.97:53904) 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.437+0000 7fa706167700 1 -- 172.21.15.97:0/3047440397 learned_addr learned my addr 172.21.15.97:0/3047440397 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa706167700 1 -- 172.21.15.97:0/3047440397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f0005040 con 0x7fa7000f5410 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa706167700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa6f0005010 tx=0x7fa6f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f0009320 con 0x7fa7000f5410 2024-02-19T22:17:57.445 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa700105dd0 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6f0013380 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6f0014e50 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.438+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa700106260 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.439+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6f00263a0 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.439+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa700045ad0 con 0x7fa7000f5410 2024-02-19T22:17:57.446 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.439+0000 7fa6f6ffd700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 0x7fa6ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:57.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.439+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6f0040e70 con 0x7fa7000f5410 2024-02-19T22:17:57.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.439+0000 7fa705966700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 0x7fa6ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:57.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.440+0000 7fa705966700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 0x7fa6ec0323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa6fc0096a0 tx=0x7fa6fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:57.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.444+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6f001ddc0 con 0x7fa7000f5410 2024-02-19T22:17:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.653+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7001070b0 con 0x7fa7000f5410 2024-02-19T22:17:57.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.653+0000 7fa6f6ffd700 1 -- 172.21.15.97:0/3047440397 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6f001d9d0 con 0x7fa7000f5410 2024-02-19T22:17:57.656 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:17:57.656 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 msgr2=0x7fa6ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 0x7fa6ec0323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fa6fc0096a0 tx=0x7fa6fc006b40).stop 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 msgr2=0x7fa7001056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fa6f0005010 tx=0x7fa6f000b040).stop 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 shutdown_connections 2024-02-19T22:17:57.657 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6ec02ff10 0x7fa6ec0323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 --2- 172.21.15.97:0/3047440397 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa7000f5410 0x7fa7001056e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 >> 172.21.15.97:0/3047440397 conn(0x7fa7000f0b70 msgr2=0x7fa7000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 shutdown_connections 2024-02-19T22:17:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:17:57.655+0000 7fa7083cb700 1 -- 172.21.15.97:0/3047440397 wait complete. 2024-02-19T22:17:57.658 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:17:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:55.818048+0000 mgr.smithi045.bvlamo (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:58.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:17:57.654561+0000 mon.smithi045 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.97:0/3047440397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:59.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:59.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:59.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:17:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:57.818351+0000 mgr.smithi045.bvlamo (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- 172.21.15.97:0/1071271218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 msgr2=0x7f92840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 --2- 172.21.15.97:0/1071271218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f92840f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f9278009fb0 tx=0x7f9278009b30).stop 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- 172.21.15.97:0/1071271218 shutdown_connections 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 --2- 172.21.15.97:0/1071271218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f92840f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- 172.21.15.97:0/1071271218 >> 172.21.15.97:0/1071271218 conn(0x7f92840f0b70 msgr2=0x7f92840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- 172.21.15.97:0/1071271218 shutdown_connections 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- 172.21.15.97:0/1071271218 wait complete. 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 Processor -- start 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.220+0000 7f928afa4700 1 -- start start 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f928afa4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f928afa4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284060220 con 0x7f92840f5410 2024-02-19T22:18:01.223 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f9288d40700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:01.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f9288d40700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53918/0 (socket says 172.21.15.97:53918) 2024-02-19T22:18:01.224 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f9288d40700 1 -- 172.21.15.97:0/599367976 learned_addr learned my addr 172.21.15.97:0/599367976 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:01.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f9288d40700 1 -- 172.21.15.97:0/599367976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9278005040 con 0x7f92840f5410 2024-02-19T22:18:01.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f9288d40700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9278005010 tx=0x7f927800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:01.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.221+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9278009320 con 0x7f92840f5410 2024-02-19T22:18:01.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9284060420 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9278013380 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9278014e50 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9284109bb0 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9278014420 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9284059100 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.222+0000 7f92817fa700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 0x7f92740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.223+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9278040f40 con 0x7f92840f5410 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.223+0000 7f9283fff700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 0x7f92740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:01.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.223+0000 7f9283fff700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 0x7f92740323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f92700096a0 tx=0x7f9270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:01.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.227+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92780095f0 con 0x7f92840f5410 2024-02-19T22:18:01.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.434+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9284045ad0 con 0x7f92840f5410 2024-02-19T22:18:01.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.435+0000 7f92817fa700 1 -- 172.21.15.97:0/599367976 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f927801db70 con 0x7f92840f5410 2024-02-19T22:18:01.437 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:01.437 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:01.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 msgr2=0x7f92740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 0x7f92740323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f92700096a0 tx=0x7f9270006b40).stop 2024-02-19T22:18:01.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 msgr2=0x7f9284109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:01.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f9278005010 tx=0x7f927800b040).stop 2024-02-19T22:18:01.438 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 shutdown_connections 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f927402ff10 0x7f92740323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 --2- 172.21.15.97:0/599367976 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92840f5410 0x7f9284109500 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 >> 172.21.15.97:0/599367976 conn(0x7f92840f0b70 msgr2=0x7f92840f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 shutdown_connections 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:01.436+0000 7f928afa4700 1 -- 172.21.15.97:0/599367976 wait complete. 2024-02-19T22:18:01.439 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:17:59.818641+0000 mgr.smithi045.bvlamo (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:01.435819+0000 mon.smithi045 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.97:0/599367976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:03.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:03.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:03.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:01.819008+0000 mgr.smithi045.bvlamo (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:03.933067+0000 mon.smithi045 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:18:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:03.934259+0000 mon.smithi045 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:18:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 -- 172.21.15.97:0/1664475809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 msgr2=0x7f69d00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 --2- 172.21.15.97:0/1664475809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f9150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f69c0009fb0 tx=0x7f69c0009b30).stop 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 -- 172.21.15.97:0/1664475809 shutdown_connections 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 --2- 172.21.15.97:0/1664475809 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f9150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 -- 172.21.15.97:0/1664475809 >> 172.21.15.97:0/1664475809 conn(0x7f69d00f0b70 msgr2=0x7f69d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 -- 172.21.15.97:0/1664475809 shutdown_connections 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.994+0000 7f69d66b3700 1 -- 172.21.15.97:0/1664475809 wait complete. 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69d66b3700 1 Processor -- start 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69d66b3700 1 -- start start 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69d66b3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69d66b3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d00f5050 con 0x7f69d0060220 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69cffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69cffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53934/0 (socket says 172.21.15.97:53934) 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.995+0000 7f69cffff700 1 -- 172.21.15.97:0/348440286 learned_addr learned my addr 172.21.15.97:0/348440286 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69cffff700 1 -- 172.21.15.97:0/348440286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c0005040 con 0x7f69d0060220 2024-02-19T22:18:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69cffff700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f69c00089f0 tx=0x7f69c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c0009320 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d00f51f0 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69c0013380 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69c0014e20 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.996+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d00f5680 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.997+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69c00095f0 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.997+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d0059100 con 0x7f69d0060220 2024-02-19T22:18:05.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.997+0000 7f69ccff9700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:05.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.997+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69c0040f50 con 0x7f69d0060220 2024-02-19T22:18:05.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.997+0000 7f69cf7fe700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:05.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:04.998+0000 7f69cf7fe700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 0x7f69b80323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:05.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.002+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69c0014420 con 0x7f69d0060220 2024-02-19T22:18:05.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.205+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69d00fc090 con 0x7f69d0060220 2024-02-19T22:18:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.206+0000 7f69ccff9700 1 -- 172.21.15.97:0/348440286 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f69c001d980 con 0x7f69d0060220 2024-02-19T22:18:05.208 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:05.208 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:05.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 msgr2=0x7f69b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:05.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 0x7f69b80323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f69c40096a0 tx=0x7f69c4006b40).stop 2024-02-19T22:18:05.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 msgr2=0x7f69d00f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:05.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f69c00089f0 tx=0x7f69c000b040).stop 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 shutdown_connections 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.207+0000 7f69d66b3700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f69b802ff10 0x7f69b80323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.208+0000 7f69d66b3700 1 --2- 172.21.15.97:0/348440286 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69d0060220 0x7f69d00f6bd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.208+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 >> 172.21.15.97:0/348440286 conn(0x7f69d00f0b70 msgr2=0x7f69d00f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.208+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 shutdown_connections 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:05.208+0000 7f69d66b3700 1 -- 172.21.15.97:0/348440286 wait complete. 2024-02-19T22:18:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:03.819233+0000 mgr.smithi045.bvlamo (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:05.207147+0000 mon.smithi045 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.97:0/348440286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:07.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:07.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:05.819609+0000 mgr.smithi045.bvlamo (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 -- 172.21.15.97:0/2760842894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 msgr2=0x7f7a400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 --2- 172.21.15.97:0/2760842894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a400f55c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f7a30009fb0 tx=0x7f7a30009b30).stop 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 -- 172.21.15.97:0/2760842894 shutdown_connections 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 --2- 172.21.15.97:0/2760842894 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a400f55c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 -- 172.21.15.97:0/2760842894 >> 172.21.15.97:0/2760842894 conn(0x7f7a400f09a0 msgr2=0x7f7a400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:09.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 -- 172.21.15.97:0/2760842894 shutdown_connections 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 -- 172.21.15.97:0/2760842894 wait complete. 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.085+0000 7f7a450ab700 1 Processor -- start 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a450ab700 1 -- start start 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a450ab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a450ab700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40108cc0 con 0x7f7a400f51a0 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a3ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a3ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35260/0 (socket says 172.21.15.97:35260) 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a3ffff700 1 -- 172.21.15.97:0/1232418920 learned_addr learned my addr 172.21.15.97:0/1232418920 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a3ffff700 1 -- 172.21.15.97:0/1232418920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30005040 con 0x7f7a400f51a0 2024-02-19T22:18:09.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.086+0000 7f7a3ffff700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7a300089f0 tx=0x7f7a3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:09.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a30009320 con 0x7f7a400f51a0 2024-02-19T22:18:09.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a40108ec0 con 0x7f7a400f51a0 2024-02-19T22:18:09.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a30013380 con 0x7f7a400f51a0 2024-02-19T22:18:09.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a30014e20 con 0x7f7a400f51a0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a40109350 con 0x7f7a400f51a0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.087+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a300095f0 con 0x7f7a400f51a0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.088+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a40045ad0 con 0x7f7a400f51a0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.088+0000 7f7a3cff9700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 0x7f7a280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.088+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a30040f10 con 0x7f7a400f51a0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.088+0000 7f7a3f7fe700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 0x7f7a280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:09.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.088+0000 7f7a3f7fe700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 0x7f7a280323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7a340096a0 tx=0x7f7a34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:09.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.093+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a30014420 con 0x7f7a400f51a0 2024-02-19T22:18:09.301 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.300+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a4010a200 con 0x7f7a400f51a0 2024-02-19T22:18:09.302 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.300+0000 7f7a3cff9700 1 -- 172.21.15.97:0/1232418920 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a3001d980 con 0x7f7a400f51a0 2024-02-19T22:18:09.303 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:09.303 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:09.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 msgr2=0x7f7a280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 0x7f7a280323d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7a340096a0 tx=0x7f7a34006b40).stop 2024-02-19T22:18:09.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 msgr2=0x7f7a401087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:09.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f7a300089f0 tx=0x7f7a3000b040).stop 2024-02-19T22:18:09.305 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 shutdown_connections 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a2802ff10 0x7f7a280323d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 --2- 172.21.15.97:0/1232418920 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a400f51a0 0x7f7a401087d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 >> 172.21.15.97:0/1232418920 conn(0x7f7a400f09a0 msgr2=0x7f7a4005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 shutdown_connections 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:09.302+0000 7f7a450ab700 1 -- 172.21.15.97:0/1232418920 wait complete. 2024-02-19T22:18:09.306 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:09.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:07.820012+0000 mgr.smithi045.bvlamo (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:09.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:09.301291+0000 mon.smithi045 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.97:0/1232418920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:10.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:10.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:11.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:09.820221+0000 mgr.smithi045.bvlamo (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:12.691 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- 172.21.15.97:0/2292791851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 msgr2=0x7f06180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 --2- 172.21.15.97:0/2292791851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f06180f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f0600009fb0 tx=0x7f0600009b30).stop 2024-02-19T22:18:12.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- 172.21.15.97:0/2292791851 shutdown_connections 2024-02-19T22:18:12.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 --2- 172.21.15.97:0/2292791851 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f06180f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- 172.21.15.97:0/2292791851 >> 172.21.15.97:0/2292791851 conn(0x7f06180f1c70 msgr2=0x7f06180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- 172.21.15.97:0/2292791851 shutdown_connections 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- 172.21.15.97:0/2292791851 wait complete. 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 Processor -- start 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.688+0000 7f061e704700 1 -- start start 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f061e704700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f061e704700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618069e20 con 0x7f06180f6510 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0617fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0617fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35288/0 (socket says 172.21.15.97:35288) 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0617fff700 1 -- 172.21.15.97:0/3843546855 learned_addr learned my addr 172.21.15.97:0/3843546855 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0617fff700 1 -- 172.21.15.97:0/3843546855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0600005040 con 0x7f06180f6510 2024-02-19T22:18:12.722 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0617fff700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0600005010 tx=0x7f060000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0600009320 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0618069fc0 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0600013380 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.689+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0600014e50 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.690+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061806a450 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.690+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0600014420 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.690+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0618059100 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.690+0000 7f0614ff9700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 0x7f06040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.690+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0600040eb0 con 0x7f06180f6510 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.691+0000 7f06177fe700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 0x7f06040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.691+0000 7f06177fe700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 0x7f06040323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f06080096a0 tx=0x7f0608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:12.724 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.695+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06000095f0 con 0x7f06180f6510 2024-02-19T22:18:12.901 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.899+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06180fc0a0 con 0x7f06180f6510 2024-02-19T22:18:12.902 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.900+0000 7f0614ff9700 1 -- 172.21.15.97:0/3843546855 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f060001db70 con 0x7f06180f6510 2024-02-19T22:18:12.902 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:12.902 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:12.903 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.901+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 msgr2=0x7f06040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 0x7f06040323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f06080096a0 tx=0x7f0608006b40).stop 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 msgr2=0x7f061806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f0600005010 tx=0x7f060000b040).stop 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 shutdown_connections 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f060402ff10 0x7f06040323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 --2- 172.21.15.97:0/3843546855 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f06180f6510 0x7f061806b9a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 >> 172.21.15.97:0/3843546855 conn(0x7f06180f1c70 msgr2=0x7f06180f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 shutdown_connections 2024-02-19T22:18:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:12.902+0000 7f061e704700 1 -- 172.21.15.97:0/3843546855 wait complete. 2024-02-19T22:18:12.905 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:11.820529+0000 mgr.smithi045.bvlamo (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:12.900808+0000 mon.smithi045 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.97:0/3843546855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:14.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:14.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:13.820871+0000 mgr.smithi045.bvlamo (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:16.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 -- 172.21.15.97:0/284888517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 msgr2=0x7fcb080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.439 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/284888517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb080f5830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7fcaf8009fb0 tx=0x7fcaf8009b30).stop 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 -- 172.21.15.97:0/284888517 shutdown_connections 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/284888517 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb080f5830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 -- 172.21.15.97:0/284888517 >> 172.21.15.97:0/284888517 conn(0x7fcb080f0b70 msgr2=0x7fcb080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 -- 172.21.15.97:0/284888517 shutdown_connections 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.437+0000 7fcb0ea26700 1 -- 172.21.15.97:0/284888517 wait complete. 2024-02-19T22:18:16.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.438+0000 7fcb0ea26700 1 Processor -- start 2024-02-19T22:18:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.438+0000 7fcb0ea26700 1 -- start start 2024-02-19T22:18:16.441 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.438+0000 7fcb0ea26700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.438+0000 7fcb0ea26700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb08060280 con 0x7fcb080f5410 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.438+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb07fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56482/0 (socket says 172.21.15.97:56482) 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb07fff700 1 -- 172.21.15.97:0/356996735 learned_addr learned my addr 172.21.15.97:0/356996735 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb07fff700 1 -- 172.21.15.97:0/356996735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8005040 con 0x7fcb080f5410 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb07fff700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcaf8005010 tx=0x7fcaf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaf8009320 con 0x7fcb080f5410 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcaf8013380 con 0x7fcb080f5410 2024-02-19T22:18:16.442 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb08060480 con 0x7fcb080f5410 2024-02-19T22:18:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcaf8014e50 con 0x7fcb080f5410 2024-02-19T22:18:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.439+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb08109c10 con 0x7fcb080f5410 2024-02-19T22:18:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.440+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcaf80095f0 con 0x7fcb080f5410 2024-02-19T22:18:16.443 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.440+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb08045ad0 con 0x7fcb080f5410 2024-02-19T22:18:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.440+0000 7fcb04ff9700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 0x7fcaf00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.441+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcaf8040e90 con 0x7fcb080f5410 2024-02-19T22:18:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.441+0000 7fcb077fe700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 0x7fcaf00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:16.444 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.441+0000 7fcb077fe700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 0x7fcaf00323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcafc0096a0 tx=0x7fcafc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:16.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.445+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcaf8014420 con 0x7fcb080f5410 2024-02-19T22:18:16.651 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.649+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb080fafa0 con 0x7fcb080f5410 2024-02-19T22:18:16.652 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.650+0000 7fcb04ff9700 1 -- 172.21.15.97:0/356996735 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcaf8010230 con 0x7fcb080f5410 2024-02-19T22:18:16.652 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:16.652 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 msgr2=0x7fcaf00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 0x7fcaf00323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fcafc0096a0 tx=0x7fcafc006b40).stop 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 msgr2=0x7fcb081094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fcaf8005010 tx=0x7fcaf800b040).stop 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 shutdown_connections 2024-02-19T22:18:16.653 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcaf002ff10 0x7fcaf00323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 --2- 172.21.15.97:0/356996735 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcb080f5410 0x7fcb081094e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 >> 172.21.15.97:0/356996735 conn(0x7fcb080f0b70 msgr2=0x7fcb080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 shutdown_connections 2024-02-19T22:18:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:16.651+0000 7fcb0ea26700 1 -- 172.21.15.97:0/356996735 wait complete. 2024-02-19T22:18:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:15.821199+0000 mgr.smithi045.bvlamo (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:16.650849+0000 mon.smithi045 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.97:0/356996735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:18.398 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:18.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:19.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:17.821442+0000 mgr.smithi045.bvlamo (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 -- 172.21.15.97:0/1762959292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 msgr2=0x7f33600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 --2- 172.21.15.97:0/1762959292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f3350009fb0 tx=0x7f3350009b30).stop 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 -- 172.21.15.97:0/1762959292 shutdown_connections 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 --2- 172.21.15.97:0/1762959292 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 -- 172.21.15.97:0/1762959292 >> 172.21.15.97:0/1762959292 conn(0x7f33600f0b70 msgr2=0x7f33600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 -- 172.21.15.97:0/1762959292 shutdown_connections 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.239+0000 7f3367eae700 1 -- 172.21.15.97:0/1762959292 wait complete. 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3367eae700 1 Processor -- start 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3367eae700 1 -- start start 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3367eae700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3367eae700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600fb960 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3365c4a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3365c4a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56488/0 (socket says 172.21.15.97:56488) 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3365c4a700 1 -- 172.21.15.97:0/579634092 learned_addr learned my addr 172.21.15.97:0/579634092 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3365c4a700 1 -- 172.21.15.97:0/579634092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3350005040 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.240+0000 7f3365c4a700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f3350005010 tx=0x7f335000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3350009320 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33600fbb00 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3350013380 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3350014e50 con 0x7f33600f5410 2024-02-19T22:18:20.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33600fbf90 con 0x7f33600f5410 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f33500095f0 con 0x7f33600f5410 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.241+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3360059100 con 0x7f33600f5410 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.242+0000 7f33567fc700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 0x7f334c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.242+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3350040e90 con 0x7f33600f5410 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.242+0000 7f3365449700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 0x7f334c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:20.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.242+0000 7f3365449700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 0x7f334c0323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:20.248 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.247+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3350014420 con 0x7f33600f5410 2024-02-19T22:18:20.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.448+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f33600fafa0 con 0x7f33600f5410 2024-02-19T22:18:20.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.449+0000 7f33567fc700 1 -- 172.21.15.97:0/579634092 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3350010230 con 0x7f33600f5410 2024-02-19T22:18:20.451 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:20.451 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 msgr2=0x7f334c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 0x7f334c0323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f335c0096a0 tx=0x7f335c006b40).stop 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 msgr2=0x7f33600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f3350005010 tx=0x7f335000b040).stop 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 shutdown_connections 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f334c02ff10 0x7f334c0323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 --2- 172.21.15.97:0/579634092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f33600f5410 0x7f33600fd4e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.450+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 >> 172.21.15.97:0/579634092 conn(0x7f33600f0b70 msgr2=0x7f336005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.451+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 shutdown_connections 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:20.451+0000 7f3367eae700 1 -- 172.21.15.97:0/579634092 wait complete. 2024-02-19T22:18:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:20.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:20.450047+0000 mon.smithi045 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.97:0/579634092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:21.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:19.821756+0000 mgr.smithi045.bvlamo (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:22.178 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:22.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:21.822052+0000 mgr.smithi045.bvlamo (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 -- 172.21.15.97:0/3342189039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 msgr2=0x7f6e380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 --2- 172.21.15.97:0/3342189039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e380f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f6e2c009fb0 tx=0x7f6e2c009b30).stop 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 -- 172.21.15.97:0/3342189039 shutdown_connections 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 --2- 172.21.15.97:0/3342189039 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e380f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 -- 172.21.15.97:0/3342189039 >> 172.21.15.97:0/3342189039 conn(0x7f6e380f0b70 msgr2=0x7f6e380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 -- 172.21.15.97:0/3342189039 shutdown_connections 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.072+0000 7f6e40117700 1 -- 172.21.15.97:0/3342189039 wait complete. 2024-02-19T22:18:24.076 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e40117700 1 Processor -- start 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e40117700 1 -- start start 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e40117700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e40117700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e38060280 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e3deb3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e3deb3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56508/0 (socket says 172.21.15.97:56508) 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e3deb3700 1 -- 172.21.15.97:0/679085097 learned_addr learned my addr 172.21.15.97:0/679085097 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.073+0000 7f6e3deb3700 1 -- 172.21.15.97:0/679085097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2c005040 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e3deb3700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6e2c005010 tx=0x7f6e2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2c009320 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e38060480 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e2c013380 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6e2c014e50 con 0x7f6e380f5410 2024-02-19T22:18:24.077 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.074+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e38109c10 con 0x7f6e380f5410 2024-02-19T22:18:24.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6e2c0095f0 con 0x7f6e380f5410 2024-02-19T22:18:24.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e38045ad0 con 0x7f6e380f5410 2024-02-19T22:18:24.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e2a7fc700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:24.078 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6e2c040e90 con 0x7f6e380f5410 2024-02-19T22:18:24.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e3d6b2700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:24.079 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.075+0000 7f6e3d6b2700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 0x7f6e240323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f6e340096a0 tx=0x7f6e34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:24.081 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.080+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6e2c014420 con 0x7f6e380f5410 2024-02-19T22:18:24.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.286+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e380fafa0 con 0x7f6e380f5410 2024-02-19T22:18:24.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.286+0000 7f6e2a7fc700 1 -- 172.21.15.97:0/679085097 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6e2c010230 con 0x7f6e380f5410 2024-02-19T22:18:24.288 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:24.288 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 msgr2=0x7f6e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 0x7f6e240323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f6e340096a0 tx=0x7f6e34006b40).stop 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 msgr2=0x7f6e381094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6e2c005010 tx=0x7f6e2c00b040).stop 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 shutdown_connections 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6e2402ff10 0x7f6e240323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 --2- 172.21.15.97:0/679085097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6e380f5410 0x7f6e381094e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 >> 172.21.15.97:0/679085097 conn(0x7f6e380f0b70 msgr2=0x7f6e380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 shutdown_connections 2024-02-19T22:18:24.290 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:24.288+0000 7f6e40117700 1 -- 172.21.15.97:0/679085097 wait complete. 2024-02-19T22:18:24.291 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:24.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:24.287353+0000 mon.smithi045 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.97:0/679085097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:25.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:23.822372+0000 mgr.smithi045.bvlamo (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:26.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:26.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:26.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:25.822675+0000 mgr.smithi045.bvlamo (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- 172.21.15.97:0/2849538215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 msgr2=0x7fcccc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 --2- 172.21.15.97:0/2849538215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc0f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fccbc009fb0 tx=0x7fccbc009b30).stop 2024-02-19T22:18:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- 172.21.15.97:0/2849538215 shutdown_connections 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 --2- 172.21.15.97:0/2849538215 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc0f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- 172.21.15.97:0/2849538215 >> 172.21.15.97:0/2849538215 conn(0x7fcccc0f0b70 msgr2=0x7fcccc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- 172.21.15.97:0/2849538215 shutdown_connections 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- 172.21.15.97:0/2849538215 wait complete. 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 Processor -- start 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- start start 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.911+0000 7fccd20fc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc106cd0 con 0x7fcccc0f5410 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fcccb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fcccb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51542/0 (socket says 172.21.15.97:51542) 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fcccb7fe700 1 -- 172.21.15.97:0/3212810863 learned_addr learned my addr 172.21.15.97:0/3212810863 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:27.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fcccb7fe700 1 -- 172.21.15.97:0/3212810863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccbc005040 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fcccb7fe700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fccbc005010 tx=0x7fccbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc009320 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc106ed0 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccbc013380 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccbc014e50 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.912+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc107360 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.913+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccbc014420 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.913+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc045ad0 con 0x7fcccc0f5410 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.913+0000 7fccb3fff700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 0x7fccb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:27.936 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.913+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccbc040e60 con 0x7fcccc0f5410 2024-02-19T22:18:27.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.913+0000 7fcccaffd700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 0x7fccb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:27.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.914+0000 7fcccaffd700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 0x7fccb40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fccc00096a0 tx=0x7fccc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:27.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:27.918+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccbc0095f0 con 0x7fcccc0f5410 2024-02-19T22:18:28.128 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.126+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcccc0fafa0 con 0x7fcccc0f5410 2024-02-19T22:18:28.129 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.127+0000 7fccb3fff700 1 -- 172.21.15.97:0/3212810863 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccbc01db70 con 0x7fcccc0f5410 2024-02-19T22:18:28.129 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:28.129 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:28.130 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 msgr2=0x7fccb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 0x7fccb40323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fccc00096a0 tx=0x7fccc0006b40).stop 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 msgr2=0x7fcccc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fccbc005010 tx=0x7fccbc00b040).stop 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 shutdown_connections 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccb402ff10 0x7fccb40323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 --2- 172.21.15.97:0/3212810863 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcccc0f5410 0x7fcccc1067e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 >> 172.21.15.97:0/3212810863 conn(0x7fcccc0f0b70 msgr2=0x7fcccc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.128+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 shutdown_connections 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:28.129+0000 7fccd20fc700 1 -- 172.21.15.97:0/3212810863 wait complete. 2024-02-19T22:18:28.131 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:28.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:28.127945+0000 mon.smithi045 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.97:0/3212810863' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:29.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:29.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:27.822931+0000 mgr.smithi045.bvlamo (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 -- 172.21.15.97:0/702753450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 msgr2=0x7f92380f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 --2- 172.21.15.97:0/702753450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f92380f90f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9228009fb0 tx=0x7f9228009b30).stop 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 -- 172.21.15.97:0/702753450 shutdown_connections 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 --2- 172.21.15.97:0/702753450 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f92380f90f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 -- 172.21.15.97:0/702753450 >> 172.21.15.97:0/702753450 conn(0x7f92380f0b70 msgr2=0x7f92380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 -- 172.21.15.97:0/702753450 shutdown_connections 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.786+0000 7f923e101700 1 -- 172.21.15.97:0/702753450 wait complete. 2024-02-19T22:18:31.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f923e101700 1 Processor -- start 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f923e101700 1 -- start start 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f923e101700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f923e101700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9238106c70 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f92377fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f92377fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51562/0 (socket says 172.21.15.97:51562) 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.787+0000 7f92377fe700 1 -- 172.21.15.97:0/1073120191 learned_addr learned my addr 172.21.15.97:0/1073120191 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f92377fe700 1 -- 172.21.15.97:0/1073120191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228005040 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f92377fe700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f92280089f0 tx=0x7f922800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9228009320 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9238106e70 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9228013380 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9228014e20 con 0x7f9238060280 2024-02-19T22:18:31.791 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.788+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9238107300 con 0x7f9238060280 2024-02-19T22:18:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.789+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9228014420 con 0x7f9238060280 2024-02-19T22:18:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.789+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9238045ad0 con 0x7f9238060280 2024-02-19T22:18:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.789+0000 7f922ffff700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 0x7f92240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:31.792 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.789+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9228040f00 con 0x7f9238060280 2024-02-19T22:18:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.789+0000 7f9236ffd700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 0x7f92240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.790+0000 7f9236ffd700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 0x7f92240323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f92200096a0 tx=0x7f9220006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:31.794+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92280095f0 con 0x7f9238060280 2024-02-19T22:18:31.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:29.823171+0000 mgr.smithi045.bvlamo (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:32.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.003+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92381081b0 con 0x7f9238060280 2024-02-19T22:18:32.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.004+0000 7f922ffff700 1 -- 172.21.15.97:0/1073120191 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f922801db70 con 0x7f9238060280 2024-02-19T22:18:32.006 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:32.006 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:32.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 msgr2=0x7f92240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:32.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 0x7f92240323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f92200096a0 tx=0x7f9220006b40).stop 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 msgr2=0x7f9238106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f92280089f0 tx=0x7f922800b040).stop 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 shutdown_connections 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f922402ff10 0x7f92240323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.005+0000 7f923e101700 1 --2- 172.21.15.97:0/1073120191 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9238060280 0x7f9238106780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.006+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 >> 172.21.15.97:0/1073120191 conn(0x7f92380f0b70 msgr2=0x7f92380f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.006+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 shutdown_connections 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:32.006+0000 7f923e101700 1 -- 172.21.15.97:0/1073120191 wait complete. 2024-02-19T22:18:32.008 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:32.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:32.004840+0000 mon.smithi045 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.97:0/1073120191' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:33.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:33.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:33.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:31.823527+0000 mgr.smithi045.bvlamo (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 -- 172.21.15.97:0/2740590371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 msgr2=0x7fe72c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 --2- 172.21.15.97:0/2740590371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c0f6930 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe714009fb0 tx=0x7fe714009b30).stop 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 -- 172.21.15.97:0/2740590371 shutdown_connections 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 --2- 172.21.15.97:0/2740590371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c0f6930 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 -- 172.21.15.97:0/2740590371 >> 172.21.15.97:0/2740590371 conn(0x7fe72c0f1c70 msgr2=0x7fe72c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 -- 172.21.15.97:0/2740590371 shutdown_connections 2024-02-19T22:18:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.702+0000 7fe7308df700 1 -- 172.21.15.97:0/2740590371 wait complete. 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe7308df700 1 Processor -- start 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe7308df700 1 -- start start 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe7308df700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe7308df700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c069e20 con 0x7fe72c0f6510 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe72a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe72a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51574/0 (socket says 172.21.15.97:51574) 2024-02-19T22:18:35.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.703+0000 7fe72a59c700 1 -- 172.21.15.97:0/3759017049 learned_addr learned my addr 172.21.15.97:0/3759017049 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:35.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe72a59c700 1 -- 172.21.15.97:0/3759017049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe714005040 con 0x7fe72c0f6510 2024-02-19T22:18:35.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe72a59c700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe7140089f0 tx=0x7fe71400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe714009320 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe714013380 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c069fc0 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe71401ed80 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.704+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c06a450 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.705+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c059100 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.705+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe71401e420 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.705+0000 7fe722ffd700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 0x7fe718032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.706+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe714040fc0 con 0x7fe72c0f6510 2024-02-19T22:18:35.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.706+0000 7fe729d9b700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 0x7fe718032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:35.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.706+0000 7fe729d9b700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 0x7fe718032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe71c0096a0 tx=0x7fe71c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:35.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.711+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe7140095f0 con 0x7fe72c0f6510 2024-02-19T22:18:35.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:33.823858+0000 mgr.smithi045.bvlamo (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:35.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.915+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe72c045ad0 con 0x7fe72c0f6510 2024-02-19T22:18:35.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.916+0000 7fe722ffd700 1 -- 172.21.15.97:0/3759017049 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe71401cb10 con 0x7fe72c0f6510 2024-02-19T22:18:35.918 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:35.918 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 msgr2=0x7fe718032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 0x7fe718032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fe71c0096a0 tx=0x7fe71c006b40).stop 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 msgr2=0x7fe72c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fe7140089f0 tx=0x7fe71400b040).stop 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 shutdown_connections 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe71802ff60 0x7fe718032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 --2- 172.21.15.97:0/3759017049 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe72c0f6510 0x7fe72c06b9a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 >> 172.21.15.97:0/3759017049 conn(0x7fe72c0f1c70 msgr2=0x7fe72c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 shutdown_connections 2024-02-19T22:18:35.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:35.918+0000 7fe7308df700 1 -- 172.21.15.97:0/3759017049 wait complete. 2024-02-19T22:18:35.921 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:36.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:35.917279+0000 mon.smithi045 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.97:0/3759017049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:37.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:37.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:37.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:35.824179+0000 mgr.smithi045.bvlamo (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 -- 172.21.15.97:0/3064706582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 msgr2=0x7f9b200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 --2- 172.21.15.97:0/3064706582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b200f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f9b10009fb0 tx=0x7f9b10009b30).stop 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 -- 172.21.15.97:0/3064706582 shutdown_connections 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 --2- 172.21.15.97:0/3064706582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b200f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 -- 172.21.15.97:0/3064706582 >> 172.21.15.97:0/3064706582 conn(0x7f9b200f0b70 msgr2=0x7f9b200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:39.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 -- 172.21.15.97:0/3064706582 shutdown_connections 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.492+0000 7f9b269cc700 1 -- 172.21.15.97:0/3064706582 wait complete. 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.493+0000 7f9b269cc700 1 Processor -- start 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.493+0000 7f9b269cc700 1 -- start start 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.493+0000 7f9b269cc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.493+0000 7f9b269cc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b20106cc0 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1ffff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56630/0 (socket says 172.21.15.97:56630) 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1ffff700 1 -- 172.21.15.97:0/2749866645 learned_addr learned my addr 172.21.15.97:0/2749866645 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1ffff700 1 -- 172.21.15.97:0/2749866645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b10005040 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1ffff700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9b10005010 tx=0x7f9b1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b10009320 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b20106ec0 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b10013380 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b10014e50 con 0x7f9b200f5410 2024-02-19T22:18:39.497 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.494+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b20107350 con 0x7f9b200f5410 2024-02-19T22:18:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.495+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b10014420 con 0x7f9b200f5410 2024-02-19T22:18:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.495+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b20045ad0 con 0x7f9b200f5410 2024-02-19T22:18:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.495+0000 7f9b1cff9700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 0x7f9b0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.495+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b10040e70 con 0x7f9b200f5410 2024-02-19T22:18:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.495+0000 7f9b1f7fe700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 0x7f9b0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:39.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.496+0000 7f9b1f7fe700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 0x7f9b0c0323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:39.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.501+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b1001db70 con 0x7f9b200f5410 2024-02-19T22:18:39.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.706+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b20108200 con 0x7f9b200f5410 2024-02-19T22:18:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b1cff9700 1 -- 172.21.15.97:0/2749866645 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9b1001db70 con 0x7f9b200f5410 2024-02-19T22:18:39.709 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:39.709 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:39.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 msgr2=0x7f9b0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 0x7f9b0c0323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f9b080096a0 tx=0x7f9b08006b40).stop 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 msgr2=0x7f9b201067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9b10005010 tx=0x7f9b1000b040).stop 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 shutdown_connections 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9b0c02ff10 0x7f9b0c0323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 --2- 172.21.15.97:0/2749866645 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9b200f5410 0x7f9b201067d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.708+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 >> 172.21.15.97:0/2749866645 conn(0x7f9b200f0b70 msgr2=0x7f9b2005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.709+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 shutdown_connections 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:39.709+0000 7f9b269cc700 1 -- 172.21.15.97:0/2749866645 wait complete. 2024-02-19T22:18:39.711 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:39.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:37.824525+0000 mgr.smithi045.bvlamo (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:40.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:39.707991+0000 mon.smithi045 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.97:0/2749866645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:41.375 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:41.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:41.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:39.824917+0000 mgr.smithi045.bvlamo (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:43.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 -- 172.21.15.97:0/822390879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 msgr2=0x7f4b1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 --2- 172.21.15.97:0/822390879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c0f6930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f4b0c009fb0 tx=0x7f4b0c009b30).stop 2024-02-19T22:18:43.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 -- 172.21.15.97:0/822390879 shutdown_connections 2024-02-19T22:18:43.393 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 --2- 172.21.15.97:0/822390879 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c0f6930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 -- 172.21.15.97:0/822390879 >> 172.21.15.97:0/822390879 conn(0x7f4b1c0f1c70 msgr2=0x7f4b1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 -- 172.21.15.97:0/822390879 shutdown_connections 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.390+0000 7f4b233d3700 1 -- 172.21.15.97:0/822390879 wait complete. 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b233d3700 1 Processor -- start 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b233d3700 1 -- start start 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b233d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b233d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c106cd0 con 0x7f4b1c0f6510 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b2116f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b2116f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:56642/0 (socket says 172.21.15.97:56642) 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.391+0000 7f4b2116f700 1 -- 172.21.15.97:0/2832584566 learned_addr learned my addr 172.21.15.97:0/2832584566 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:43.394 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b2116f700 1 -- 172.21.15.97:0/2832584566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c005040 con 0x7f4b1c0f6510 2024-02-19T22:18:43.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b2116f700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f4b0c005010 tx=0x7f4b0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:43.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b0c009320 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c106ed0 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b0c013380 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b0c014e50 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.392+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c107360 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.393+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b0c0095f0 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.393+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c045ad0 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.393+0000 7f4b11ffb700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 0x7f4b080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.393+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b0c040e70 con 0x7f4b1c0f6510 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.393+0000 7f4b2096e700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 0x7f4b080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.394+0000 7f4b2096e700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 0x7f4b080323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:43.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.398+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b0c014420 con 0x7f4b1c0f6510 2024-02-19T22:18:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.605+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b1c108210 con 0x7f4b1c0f6510 2024-02-19T22:18:43.607 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.605+0000 7f4b11ffb700 1 -- 172.21.15.97:0/2832584566 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4b0c010230 con 0x7f4b1c0f6510 2024-02-19T22:18:43.608 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:43.608 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:43.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.607+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 msgr2=0x7f4b080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.607+0000 7f4b233d3700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 0x7f4b080323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f4b180096a0 tx=0x7f4b18006b40).stop 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.607+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 msgr2=0x7f4b1c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.607+0000 7f4b233d3700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f4b0c005010 tx=0x7f4b0c00b040).stop 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 shutdown_connections 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4b0802ff10 0x7f4b080323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 --2- 172.21.15.97:0/2832584566 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4b1c0f6510 0x7f4b1c1067e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 >> 172.21.15.97:0/2832584566 conn(0x7f4b1c0f1c70 msgr2=0x7f4b1c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 shutdown_connections 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:43.608+0000 7f4b233d3700 1 -- 172.21.15.97:0/2832584566 wait complete. 2024-02-19T22:18:43.610 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:43.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:41.825317+0000 mgr.smithi045.bvlamo (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:44.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:43.606427+0000 mon.smithi045 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.97:0/2832584566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:45.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:45.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:43.825641+0000 mgr.smithi045.bvlamo (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 -- 172.21.15.97:0/977244691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 msgr2=0x7fc7840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/977244691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc7840f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fc774009fb0 tx=0x7fc774009b30).stop 2024-02-19T22:18:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 -- 172.21.15.97:0/977244691 shutdown_connections 2024-02-19T22:18:47.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/977244691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc7840f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 -- 172.21.15.97:0/977244691 >> 172.21.15.97:0/977244691 conn(0x7fc7840f0b70 msgr2=0x7fc7840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 -- 172.21.15.97:0/977244691 shutdown_connections 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.278+0000 7fc78b5e3700 1 -- 172.21.15.97:0/977244691 wait complete. 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78b5e3700 1 Processor -- start 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78b5e3700 1 -- start start 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78b5e3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78b5e3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc784105c00 con 0x7fc7840f5410 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78937f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78937f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51334/0 (socket says 172.21.15.97:51334) 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.279+0000 7fc78937f700 1 -- 172.21.15.97:0/1827155048 learned_addr learned my addr 172.21.15.97:0/1827155048 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc78937f700 1 -- 172.21.15.97:0/1827155048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774005040 con 0x7fc7840f5410 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc78937f700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc7740089f0 tx=0x7fc77400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc774009320 con 0x7fc7840f5410 2024-02-19T22:18:47.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc784105e00 con 0x7fc7840f5410 2024-02-19T22:18:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc774013380 con 0x7fc7840f5410 2024-02-19T22:18:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc77401ed80 con 0x7fc7840f5410 2024-02-19T22:18:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.280+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc784106290 con 0x7fc7840f5410 2024-02-19T22:18:47.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.281+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc77401e420 con 0x7fc7840f5410 2024-02-19T22:18:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.281+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc784045ad0 con 0x7fc7840f5410 2024-02-19T22:18:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.281+0000 7fc779ffb700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 0x7fc770032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.281+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7740410c0 con 0x7fc7840f5410 2024-02-19T22:18:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.281+0000 7fc788b7e700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 0x7fc770032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:47.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.282+0000 7fc788b7e700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 0x7fc770032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc7800096a0 tx=0x7fc780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:47.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.286+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc774014cc0 con 0x7fc7840f5410 2024-02-19T22:18:47.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.490+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc784107140 con 0x7fc7840f5410 2024-02-19T22:18:47.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.491+0000 7fc779ffb700 1 -- 172.21.15.97:0/1827155048 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc77401cb10 con 0x7fc7840f5410 2024-02-19T22:18:47.493 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:47.493 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 msgr2=0x7fc770032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 0x7fc770032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fc7800096a0 tx=0x7fc780006b40).stop 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 msgr2=0x7fc784105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc7740089f0 tx=0x7fc77400b040).stop 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 shutdown_connections 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc77002ffb0 0x7fc770032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.495 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 --2- 172.21.15.97:0/1827155048 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7840f5410 0x7fc784105710 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:47.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 >> 172.21.15.97:0/1827155048 conn(0x7fc7840f0b70 msgr2=0x7fc7840f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:47.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 shutdown_connections 2024-02-19T22:18:47.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:47.493+0000 7fc78b5e3700 1 -- 172.21.15.97:0/1827155048 wait complete. 2024-02-19T22:18:47.496 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:45.825935+0000 mgr.smithi045.bvlamo (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:47.492086+0000 mon.smithi045 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.97:0/1827155048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:49.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:49.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:47.826299+0000 mgr.smithi045.bvlamo (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:51.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 -- 172.21.15.97:0/3611368102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 msgr2=0x7f2f880f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 --2- 172.21.15.97:0/3611368102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f880f9150 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f2f78009fb0 tx=0x7f2f78009b30).stop 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 -- 172.21.15.97:0/3611368102 shutdown_connections 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 --2- 172.21.15.97:0/3611368102 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f880f9150 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 -- 172.21.15.97:0/3611368102 >> 172.21.15.97:0/3611368102 conn(0x7f2f880f0b70 msgr2=0x7f2f880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 -- 172.21.15.97:0/3611368102 shutdown_connections 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.012+0000 7f2f8d123700 1 -- 172.21.15.97:0/3611368102 wait complete. 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f8d123700 1 Processor -- start 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f8d123700 1 -- start start 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f8d123700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f8d123700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f88106c90 con 0x7f2f88060220 2024-02-19T22:18:51.017 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f86d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f86d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51352/0 (socket says 172.21.15.97:51352) 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.013+0000 7f2f86d9d700 1 -- 172.21.15.97:0/449493569 learned_addr learned my addr 172.21.15.97:0/449493569 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f86d9d700 1 -- 172.21.15.97:0/449493569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f78005040 con 0x7f2f88060220 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f86d9d700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2f780089f0 tx=0x7f2f7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f78009320 con 0x7f2f88060220 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f88106e90 con 0x7f2f88060220 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f78013380 con 0x7f2f88060220 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f78014e20 con 0x7f2f88060220 2024-02-19T22:18:51.018 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.014+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f88107320 con 0x7f2f88060220 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f780095f0 con 0x7f2f88060220 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f88045ad0 con 0x7f2f88060220 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f6f7fe700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 0x7f2f700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f78040e90 con 0x7f2f88060220 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f8659c700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 0x7f2f700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:51.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.015+0000 7f2f8659c700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 0x7f2f700323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f2f7c0096a0 tx=0x7f2f7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:51.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.020+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f78014420 con 0x7f2f88060220 2024-02-19T22:18:51.229 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.227+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f881081d0 con 0x7f2f88060220 2024-02-19T22:18:51.230 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.228+0000 7f2f6f7fe700 1 -- 172.21.15.97:0/449493569 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f7801d980 con 0x7f2f88060220 2024-02-19T22:18:51.230 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:51.230 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 msgr2=0x7f2f700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 0x7f2f700323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f2f7c0096a0 tx=0x7f2f7c006b40).stop 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 msgr2=0x7f2f881067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2f780089f0 tx=0x7f2f7800b040).stop 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 shutdown_connections 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f2f7002ff10 0x7f2f700323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 --2- 172.21.15.97:0/449493569 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f2f88060220 0x7f2f881067a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 >> 172.21.15.97:0/449493569 conn(0x7f2f880f0b70 msgr2=0x7f2f880f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 shutdown_connections 2024-02-19T22:18:51.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:51.230+0000 7f2f8d123700 1 -- 172.21.15.97:0/449493569 wait complete. 2024-02-19T22:18:51.233 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:49.826715+0000 mgr.smithi045.bvlamo (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:51.229109+0000 mon.smithi045 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.97:0/449493569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:52.872 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:52.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:53.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:51.827065+0000 mgr.smithi045.bvlamo (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 -- 172.21.15.97:0/3394332501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 msgr2=0x7f61840fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3394332501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61840fb380 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f616c009fb0 tx=0x7f616c009b30).stop 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 -- 172.21.15.97:0/3394332501 shutdown_connections 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3394332501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61840fb380 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 -- 172.21.15.97:0/3394332501 >> 172.21.15.97:0/3394332501 conn(0x7f61840f09a0 msgr2=0x7f61840f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 -- 172.21.15.97:0/3394332501 shutdown_connections 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 -- 172.21.15.97:0/3394332501 wait complete. 2024-02-19T22:18:54.845 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.842+0000 7f6188fa0700 1 Processor -- start 2024-02-19T22:18:54.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f6188fa0700 1 -- start start 2024-02-19T22:18:54.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f6188fa0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f6188fa0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6184108cc0 con 0x7f6184060180 2024-02-19T22:18:54.847 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f61837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f61837fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51374/0 (socket says 172.21.15.97:51374) 2024-02-19T22:18:54.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.843+0000 7f61837fe700 1 -- 172.21.15.97:0/3903711689 learned_addr learned my addr 172.21.15.97:0/3903711689 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:54.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f61837fe700 1 -- 172.21.15.97:0/3903711689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c005040 con 0x7f6184060180 2024-02-19T22:18:54.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f61837fe700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f616c005010 tx=0x7f616c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.848 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f616c009320 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6184108ec0 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f616c013380 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f616c014e50 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.844+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6184109350 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.845+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f616c0095f0 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.845+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6184045ad0 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.845+0000 7f617bfff700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 0x7f61700323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.845+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f616c040fa0 con 0x7f6184060180 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.845+0000 7f6182ffd700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 0x7f61700323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.846+0000 7f6182ffd700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 0x7f61700323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f61740096a0 tx=0x7f6174006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:54.850+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f616c014420 con 0x7f6184060180 2024-02-19T22:18:55.057 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.056+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61840fe020 con 0x7f6184060180 2024-02-19T22:18:55.058 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.056+0000 7f617bfff700 1 -- 172.21.15.97:0/3903711689 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f616c010230 con 0x7f6184060180 2024-02-19T22:18:55.058 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:55.058 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 msgr2=0x7f61700323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 0x7f61700323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f61740096a0 tx=0x7f6174006b40).stop 2024-02-19T22:18:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 msgr2=0x7f61841087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:55.060 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f616c005010 tx=0x7f616c00b040).stop 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 shutdown_connections 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f617002ff10 0x7f61700323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 --2- 172.21.15.97:0/3903711689 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6184060180 0x7f61841087d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 >> 172.21.15.97:0/3903711689 conn(0x7f61840f09a0 msgr2=0x7f61840f1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 shutdown_connections 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:55.058+0000 7f6188fa0700 1 -- 172.21.15.97:0/3903711689 wait complete. 2024-02-19T22:18:55.061 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:55.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:53.827412+0000 mgr.smithi045.bvlamo (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:55.057436+0000 mon.smithi045 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.97:0/3903711689' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:56.678 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:56.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:57.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:55.827902+0000 mgr.smithi045.bvlamo (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:58.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.605+0000 7f6deda3a700 1 -- 172.21.15.97:0/4027423603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 msgr2=0x7f6de80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.605+0000 7f6deda3a700 1 --2- 172.21.15.97:0/4027423603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de80f4be0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f6ddc009fb0 tx=0x7f6ddc009b30).stop 2024-02-19T22:18:58.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 -- 172.21.15.97:0/4027423603 shutdown_connections 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 --2- 172.21.15.97:0/4027423603 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de80f4be0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 -- 172.21.15.97:0/4027423603 >> 172.21.15.97:0/4027423603 conn(0x7f6de80f02e0 msgr2=0x7f6de80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 -- 172.21.15.97:0/4027423603 shutdown_connections 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 -- 172.21.15.97:0/4027423603 wait complete. 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 Processor -- start 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.606+0000 7f6deda3a700 1 -- start start 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6deda3a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6deda3a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de8108ed0 con 0x7f6de80f47c0 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6de6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54096/0 (socket says 172.21.15.97:54096) 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6de6ffd700 1 -- 172.21.15.97:0/1562951516 learned_addr learned my addr 172.21.15.97:0/1562951516 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6de6ffd700 1 -- 172.21.15.97:0/1562951516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc005040 con 0x7f6de80f47c0 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6de6ffd700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6ddc005010 tx=0x7f6ddc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:58.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ddc009320 con 0x7f6de80f47c0 2024-02-19T22:18:58.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de81090d0 con 0x7f6de80f47c0 2024-02-19T22:18:58.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ddc013380 con 0x7f6de80f47c0 2024-02-19T22:18:58.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ddc014e50 con 0x7f6de80f47c0 2024-02-19T22:18:58.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.607+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de8109560 con 0x7f6de80f47c0 2024-02-19T22:18:58.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.608+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ddc014420 con 0x7f6de80f47c0 2024-02-19T22:18:58.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.608+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de8045ad0 con 0x7f6de80f47c0 2024-02-19T22:18:58.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.608+0000 7f6dcf7fe700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:58.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.608+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ddc040e30 con 0x7f6de80f47c0 2024-02-19T22:18:58.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.609+0000 7f6de67fc700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:58.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.609+0000 7f6de67fc700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 0x7f6dd00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6dd80096a0 tx=0x7f6dd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:58.615 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.613+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ddc0095f0 con 0x7f6de80f47c0 2024-02-19T22:18:58.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.819+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6de810a410 con 0x7f6de80f47c0 2024-02-19T22:18:58.822 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.820+0000 7f6dcf7fe700 1 -- 172.21.15.97:0/1562951516 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6ddc01db70 con 0x7f6de80f47c0 2024-02-19T22:18:58.822 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:18:58.822 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:58.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 msgr2=0x7f6dd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 0x7f6dd00323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6dd80096a0 tx=0x7f6dd8006b40).stop 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 msgr2=0x7f6de81089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f6ddc005010 tx=0x7f6ddc00b040).stop 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 shutdown_connections 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6dd002ff10 0x7f6dd00323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 --2- 172.21.15.97:0/1562951516 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6de80f47c0 0x7f6de81089e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.821+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 >> 172.21.15.97:0/1562951516 conn(0x7f6de80f02e0 msgr2=0x7f6de80f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.822+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 shutdown_connections 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:18:58.822+0000 7f6deda3a700 1 -- 172.21.15.97:0/1562951516 wait complete. 2024-02-19T22:18:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:18:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:57.828283+0000 mgr.smithi045.bvlamo (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:18:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:18:58.820849+0000 mon.smithi045 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.97:0/1562951516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:00.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:00.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:18:59.828644+0000 mgr.smithi045.bvlamo (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 -- 172.21.15.97:0/2834293596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 msgr2=0x7f67900f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 --2- 172.21.15.97:0/2834293596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67900f9150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f6784009fb0 tx=0x7f6784009b30).stop 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 -- 172.21.15.97:0/2834293596 shutdown_connections 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 --2- 172.21.15.97:0/2834293596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67900f9150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 -- 172.21.15.97:0/2834293596 >> 172.21.15.97:0/2834293596 conn(0x7f67900f0b70 msgr2=0x7f67900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 -- 172.21.15.97:0/2834293596 shutdown_connections 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.498+0000 7f6796a85700 1 -- 172.21.15.97:0/2834293596 wait complete. 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6796a85700 1 Processor -- start 2024-02-19T22:19:02.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6796a85700 1 -- start start 2024-02-19T22:19:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6796a85700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6796a85700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6790106c90 con 0x7f6790060220 2024-02-19T22:19:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6794821700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6794821700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54114/0 (socket says 172.21.15.97:54114) 2024-02-19T22:19:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.499+0000 7f6794821700 1 -- 172.21.15.97:0/573952386 learned_addr learned my addr 172.21.15.97:0/573952386 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f6794821700 1 -- 172.21.15.97:0/573952386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784005040 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f6794821700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f67840089f0 tx=0x7f678400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6784009320 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6790106e90 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6784013380 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6784014e20 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.500+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6790107320 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.501+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67840095f0 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.501+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6790045ad0 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.501+0000 7f678d7fa700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 0x7f6780032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.501+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6784040f00 con 0x7f6790060220 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.501+0000 7f678ffff700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 0x7f6780032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:02.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.502+0000 7f678ffff700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 0x7f6780032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:02.507 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.506+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6784014420 con 0x7f6790060220 2024-02-19T22:19:02.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.710+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67901081d0 con 0x7f6790060220 2024-02-19T22:19:02.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.711+0000 7f678d7fa700 1 -- 172.21.15.97:0/573952386 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f678401d980 con 0x7f6790060220 2024-02-19T22:19:02.713 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:02.713 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 msgr2=0x7f6780032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 0x7f6780032470 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f677c0096a0 tx=0x7f677c006b40).stop 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 msgr2=0x7f67901067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f67840089f0 tx=0x7f678400b040).stop 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 shutdown_connections 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f678002ffb0 0x7f6780032470 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 --2- 172.21.15.97:0/573952386 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6790060220 0x7f67901067a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 >> 172.21.15.97:0/573952386 conn(0x7f67900f0b70 msgr2=0x7f67900f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 shutdown_connections 2024-02-19T22:19:02.715 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:02.713+0000 7f6796a85700 1 -- 172.21.15.97:0/573952386 wait complete. 2024-02-19T22:19:02.716 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:01.829042+0000 mgr.smithi045.bvlamo (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:02.712175+0000 mon.smithi045 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.97:0/573952386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:04.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:04.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:04.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:03.934198+0000 mon.smithi045 (mon.0) 193 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:19:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:03.935196+0000 mon.smithi045 (mon.0) 194 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:19:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:03.829389+0000 mgr.smithi045.bvlamo (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2307343812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 msgr2=0x7f8fb80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2307343812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb80f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8fac009fb0 tx=0x7f8fac009b30).stop 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2307343812 shutdown_connections 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2307343812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb80f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2307343812 >> 172.21.15.97:0/2307343812 conn(0x7f8fb80f09a0 msgr2=0x7f8fb80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2307343812 shutdown_connections 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.339+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2307343812 wait complete. 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb7fff700 1 Processor -- start 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb7fff700 1 -- start start 2024-02-19T22:19:06.343 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb7fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb81059c0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54140/0 (socket says 172.21.15.97:54140) 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb6ffd700 1 -- 172.21.15.97:0/2034406276 learned_addr learned my addr 172.21.15.97:0/2034406276 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.340+0000 7f8fb6ffd700 1 -- 172.21.15.97:0/2034406276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac005040 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8fb6ffd700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8fac0089f0 tx=0x7f8fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fac009320 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb8105bc0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fac013380 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fac014e20 con 0x7f8fb80f51a0 2024-02-19T22:19:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.341+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb8106050 con 0x7f8fb80f51a0 2024-02-19T22:19:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.342+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fac0095f0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.342+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb8045ad0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.342+0000 7f8f9f7fe700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 0x7f8fa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.342+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fac040fb0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.342+0000 7f8fb67fc700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 0x7f8fa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:06.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.343+0000 7f8fb67fc700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 0x7f8fa00323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8fa80096a0 tx=0x7f8fa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:06.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.347+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fac06d2b0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.553 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.551+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fb8106ea0 con 0x7f8fb80f51a0 2024-02-19T22:19:06.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.552+0000 7f8f9f7fe700 1 -- 172.21.15.97:0/2034406276 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8fac014420 con 0x7f8fb80f51a0 2024-02-19T22:19:06.554 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:06.554 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:06.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 msgr2=0x7f8fa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 0x7f8fa00323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f8fa80096a0 tx=0x7f8fa8006b40).stop 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 msgr2=0x7f8fb81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8fac0089f0 tx=0x7f8fac00b040).stop 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 shutdown_connections 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f8fa002ff10 0x7f8fa00323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 --2- 172.21.15.97:0/2034406276 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f8fb80f51a0 0x7f8fb81054d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.553+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 >> 172.21.15.97:0/2034406276 conn(0x7f8fb80f09a0 msgr2=0x7f8fb80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.554+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 shutdown_connections 2024-02-19T22:19:06.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:06.554+0000 7f8fb7fff700 1 -- 172.21.15.97:0/2034406276 wait complete. 2024-02-19T22:19:06.557 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:06.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:06.553021+0000 mon.smithi045 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.97:0/2034406276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:05.829803+0000 mgr.smithi045.bvlamo (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:08.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:08.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:07.830208+0000 mgr.smithi045.bvlamo (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 -- 172.21.15.97:0/2665474984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 msgr2=0x7fe0200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 --2- 172.21.15.97:0/2665474984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0200f6930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe008009fb0 tx=0x7fe008009b30).stop 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 -- 172.21.15.97:0/2665474984 shutdown_connections 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 --2- 172.21.15.97:0/2665474984 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0200f6930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 -- 172.21.15.97:0/2665474984 >> 172.21.15.97:0/2665474984 conn(0x7fe0200f1c70 msgr2=0x7fe0200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 -- 172.21.15.97:0/2665474984 shutdown_connections 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.212+0000 7fe025463700 1 -- 172.21.15.97:0/2665474984 wait complete. 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe025463700 1 Processor -- start 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe025463700 1 -- start start 2024-02-19T22:19:10.225 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe025463700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe025463700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe020106cd0 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe01effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe01effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:36552/0 (socket says 172.21.15.97:36552) 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.213+0000 7fe01effd700 1 -- 172.21.15.97:0/1142606932 learned_addr learned my addr 172.21.15.97:0/1142606932 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe01effd700 1 -- 172.21.15.97:0/1142606932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe008005040 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe01effd700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe008005010 tx=0x7fe00800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe008009320 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe020106ed0 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe008013380 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe008014e50 con 0x7fe0200f6510 2024-02-19T22:19:10.226 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.214+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe020107360 con 0x7fe0200f6510 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0080263a0 con 0x7fe0200f6510 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe0177fe700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 0x7fe00c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe008040e80 con 0x7fe0200f6510 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe020045ad0 con 0x7fe0200f6510 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe01e7fc700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 0x7fe00c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.215+0000 7fe01e7fc700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 0x7fe00c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:10.227 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.220+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe008012920 con 0x7fe0200f6510 2024-02-19T22:19:10.426 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.424+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0201080b0 con 0x7fe0200f6510 2024-02-19T22:19:10.427 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.425+0000 7fe0177fe700 1 -- 172.21.15.97:0/1142606932 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe00802b020 con 0x7fe0200f6510 2024-02-19T22:19:10.427 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:10.427 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 msgr2=0x7fe00c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 0x7fe00c0323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fe0100096a0 tx=0x7fe010006b40).stop 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 msgr2=0x7fe0201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe008005010 tx=0x7fe00800b040).stop 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 shutdown_connections 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe00c02ff10 0x7fe00c0323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 --2- 172.21.15.97:0/1142606932 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe0200f6510 0x7fe0201067e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:10.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 >> 172.21.15.97:0/1142606932 conn(0x7fe0200f1c70 msgr2=0x7fe0200f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:10.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 shutdown_connections 2024-02-19T22:19:10.430 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:10.427+0000 7fe025463700 1 -- 172.21.15.97:0/1142606932 wait complete. 2024-02-19T22:19:10.430 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:10.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:10.426218+0000 mon.smithi045 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.97:0/1142606932' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:09.830512+0000 mgr.smithi045.bvlamo (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:12.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:12.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:11.830912+0000 mgr.smithi045.bvlamo (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:12.690510+0000 mon.smithi045 (mon.0) 197 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T22:19:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:12.692916+0000 mon.smithi045 (mon.0) 198 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi045", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T22:19:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:12.693726+0000 mon.smithi045 (mon.0) 199 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:19:13.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 -- 172.21.15.97:0/4063862619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 msgr2=0x7f46ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 --2- 172.21.15.97:0/4063862619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f46e0009fb0 tx=0x7f46e0009b30).stop 2024-02-19T22:19:13.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 -- 172.21.15.97:0/4063862619 shutdown_connections 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 --2- 172.21.15.97:0/4063862619 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 -- 172.21.15.97:0/4063862619 >> 172.21.15.97:0/4063862619 conn(0x7f46ec0f0b70 msgr2=0x7f46ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 -- 172.21.15.97:0/4063862619 shutdown_connections 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.960+0000 7f46f19c1700 1 -- 172.21.15.97:0/4063862619 wait complete. 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46f19c1700 1 Processor -- start 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46f19c1700 1 -- start start 2024-02-19T22:19:13.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46f19c1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46f19c1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec0fb960 con 0x7f46ec0f5410 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46eaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46eaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:36564/0 (socket says 172.21.15.97:36564) 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46eaffd700 1 -- 172.21.15.97:0/2185799335 learned_addr learned my addr 172.21.15.97:0/2185799335 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.961+0000 7f46eaffd700 1 -- 172.21.15.97:0/2185799335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e0005040 con 0x7f46ec0f5410 2024-02-19T22:19:13.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46eaffd700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f46e0005010 tx=0x7f46e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46e0009320 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec0fbb00 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec0fbf90 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46e0013380 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.962+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46e0014e50 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec059100 con 0x7f46ec0f5410 2024-02-19T22:19:13.968 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46e00095f0 con 0x7f46ec0f5410 2024-02-19T22:19:13.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46d37fe700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 0x7f46d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46e0040ff0 con 0x7f46ec0f5410 2024-02-19T22:19:13.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46ea7fc700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 0x7f46d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.963+0000 7f46ea7fc700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 0x7f46d40323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f46dc0096a0 tx=0x7f46dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.969 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:13.968+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46e0014420 con 0x7f46ec0f5410 2024-02-19T22:19:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.170+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46ec045ad0 con 0x7f46ec0f5410 2024-02-19T22:19:14.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.170+0000 7f46d37fe700 1 -- 172.21.15.97:0/2185799335 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46e0010230 con 0x7f46ec0f5410 2024-02-19T22:19:14.173 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:14.173 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 msgr2=0x7f46d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 0x7f46d40323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f46dc0096a0 tx=0x7f46dc006b40).stop 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 msgr2=0x7f46ec0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f46e0005010 tx=0x7f46e000b040).stop 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 shutdown_connections 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46d402ff10 0x7f46d40323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 --2- 172.21.15.97:0/2185799335 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46ec0f5410 0x7f46ec0fd4e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 >> 172.21.15.97:0/2185799335 conn(0x7f46ec0f0b70 msgr2=0x7f46ec05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:14.174 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 shutdown_connections 2024-02-19T22:19:14.175 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:14.172+0000 7f46f19c1700 1 -- 172.21.15.97:0/2185799335 wait complete. 2024-02-19T22:19:14.175 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:19:12.689265+0000 mgr.smithi045.bvlamo (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi045 ... 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:14:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:15:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:16:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:19:14.744 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:17:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:17:57Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:18:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:19:14.745 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('uid/gid not found') 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: uid/gid not found 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: yield (conn, connr) 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: code, '\n'.join(err))) 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi045 ... 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:14:51Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.746 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:15:28Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:16:06Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:17:20Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:17:57Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:18:35Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:19:14.747 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('uid/gid not found') 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: uid/gid not found 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:19:12.694478+0000 mgr.smithi045.bvlamo (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi045 on smithi045 2024-02-19T22:19:14.748 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:14.171528+0000 mon.smithi045 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.97:0/2185799335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:15.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:13.831132+0000 mgr.smithi045.bvlamo (mgr.14162) 158 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.549600+0000 mon.smithi045 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:19:15.894 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:15.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.686993+0000 mon.smithi045 (mon.0) 202 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.688767+0000 mon.smithi045 (mon.0) 203 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.689157+0000 mon.smithi045 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.689583+0000 mgr.smithi045.bvlamo (mgr.14162) 159 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:15.691809+0000 mon.smithi045 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:19:15.694604+0000 mgr.smithi045.bvlamo (mgr.14162) 160 : cephadm [INF] Deploying daemon grafana.smithi045 on smithi045 2024-02-19T22:19:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:15.831377+0000 mgr.smithi045.bvlamo (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:17.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- 172.21.15.97:0/420565306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 msgr2=0x7f31ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 --2- 172.21.15.97:0/420565306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac0f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f31a8009fb0 tx=0x7f31a8009b30).stop 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- 172.21.15.97:0/420565306 shutdown_connections 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 --2- 172.21.15.97:0/420565306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac0f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- 172.21.15.97:0/420565306 >> 172.21.15.97:0/420565306 conn(0x7f31ac0f09a0 msgr2=0x7f31ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- 172.21.15.97:0/420565306 shutdown_connections 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- 172.21.15.97:0/420565306 wait complete. 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 Processor -- start 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.749+0000 7f31b22e7700 1 -- start start 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b22e7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b22e7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac1059c0 con 0x7f31ac0f51a0 2024-02-19T22:19:17.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b12e5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b12e5700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54320/0 (socket says 172.21.15.97:54320) 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b12e5700 1 -- 172.21.15.97:0/3642682799 learned_addr learned my addr 172.21.15.97:0/3642682799 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b12e5700 1 -- 172.21.15.97:0/3642682799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a8005040 con 0x7f31ac0f51a0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b12e5700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f31a80089f0 tx=0x7f31a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a8009320 con 0x7f31ac0f51a0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ac105bc0 con 0x7f31ac0f51a0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31a8013380 con 0x7f31ac0f51a0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.750+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f31a8014e20 con 0x7f31ac0f51a0 2024-02-19T22:19:17.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.751+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ac106050 con 0x7f31ac0f51a0 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.751+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31a80095f0 con 0x7f31ac0f51a0 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.751+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ac045ad0 con 0x7f31ac0f51a0 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.752+0000 7f31a1ffb700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 0x7f3198032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.752+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31a8040ec0 con 0x7f31ac0f51a0 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.752+0000 7f31b0ae4700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 0x7f3198032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:17.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.752+0000 7f31b0ae4700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 0x7f3198032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f319c0096a0 tx=0x7f319c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:17.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.756+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f31a8014420 con 0x7f31ac0f51a0 2024-02-19T22:19:17.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.960+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31ac106f00 con 0x7f31ac0f51a0 2024-02-19T22:19:17.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.961+0000 7f31a1ffb700 1 -- 172.21.15.97:0/3642682799 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f31a8010230 con 0x7f31ac0f51a0 2024-02-19T22:19:17.964 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:17.964 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 msgr2=0x7f3198032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 0x7f3198032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f319c0096a0 tx=0x7f319c006b40).stop 2024-02-19T22:19:17.965 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 msgr2=0x7f31ac1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f31a80089f0 tx=0x7f31a800b040).stop 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 shutdown_connections 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f319802ff60 0x7f3198032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 --2- 172.21.15.97:0/3642682799 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f31ac0f51a0 0x7f31ac1054d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 >> 172.21.15.97:0/3642682799 conn(0x7f31ac0f09a0 msgr2=0x7f31ac0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:17.966 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 shutdown_connections 2024-02-19T22:19:17.967 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:17.963+0000 7f31b22e7700 1 -- 172.21.15.97:0/3642682799 wait complete. 2024-02-19T22:19:17.967 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:18.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:17.962092+0000 mon.smithi045 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.97:0/3642682799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:19.579 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:19.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:19.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:17.831718+0000 mgr.smithi045.bvlamo (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- 172.21.15.97:0/2787845556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 msgr2=0x7f023c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 --2- 172.21.15.97:0/2787845556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c0f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f0224009fb0 tx=0x7f0224009b30).stop 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- 172.21.15.97:0/2787845556 shutdown_connections 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 --2- 172.21.15.97:0/2787845556 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c0f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- 172.21.15.97:0/2787845556 >> 172.21.15.97:0/2787845556 conn(0x7f023c0f1c70 msgr2=0x7f023c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- 172.21.15.97:0/2787845556 shutdown_connections 2024-02-19T22:19:21.593 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- 172.21.15.97:0/2787845556 wait complete. 2024-02-19T22:19:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 Processor -- start 2024-02-19T22:19:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.591+0000 7f0240e81700 1 -- start start 2024-02-19T22:19:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f0240e81700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f0240e81700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c106cd0 con 0x7f023c0f6510 2024-02-19T22:19:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f023a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f023a59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54344/0 (socket says 172.21.15.97:54344) 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f023a59c700 1 -- 172.21.15.97:0/152136238 learned_addr learned my addr 172.21.15.97:0/152136238 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.592+0000 7f023a59c700 1 -- 172.21.15.97:0/152136238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0224005040 con 0x7f023c0f6510 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f023a59c700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0224005010 tx=0x7f022400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0224009320 con 0x7f023c0f6510 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c106ed0 con 0x7f023c0f6510 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c107360 con 0x7f023c0f6510 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0224013380 con 0x7f023c0f6510 2024-02-19T22:19:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.593+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0224014df0 con 0x7f023c0f6510 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02240095f0 con 0x7f023c0f6510 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023c045ad0 con 0x7f023c0f6510 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0232ffd700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 0x7f02280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0224040e90 con 0x7f023c0f6510 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0239d9b700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 0x7f02280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:21.597 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.594+0000 7f0239d9b700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 0x7f02280323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f022c0096a0 tx=0x7f022c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:21.600 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.599+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0224014420 con 0x7f023c0f6510 2024-02-19T22:19:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.806+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f023c108210 con 0x7f023c0f6510 2024-02-19T22:19:21.808 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.807+0000 7f0232ffd700 1 -- 172.21.15.97:0/152136238 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f022401d9d0 con 0x7f023c0f6510 2024-02-19T22:19:21.809 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:21.809 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:21.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 msgr2=0x7f02280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 0x7f02280323d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f022c0096a0 tx=0x7f022c006b40).stop 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 msgr2=0x7f023c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0224005010 tx=0x7f022400b040).stop 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 shutdown_connections 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f022802ff10 0x7f02280323d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 --2- 172.21.15.97:0/152136238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f023c0f6510 0x7f023c1067e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 >> 172.21.15.97:0/152136238 conn(0x7f023c0f1c70 msgr2=0x7f023c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.808+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 shutdown_connections 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:21.809+0000 7f0240e81700 1 -- 172.21.15.97:0/152136238 wait complete. 2024-02-19T22:19:21.811 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:21.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:19.831995+0000 mgr.smithi045.bvlamo (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:22.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:21.807871+0000 mon.smithi045 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.97:0/152136238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:23.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:23.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:21.832248+0000 mgr.smithi045.bvlamo (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 -- 172.21.15.97:0/2927011993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 msgr2=0x7fcf100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2927011993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf100f5830 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fcf0c009fb0 tx=0x7fcf0c009b30).stop 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 -- 172.21.15.97:0/2927011993 shutdown_connections 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2927011993 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf100f5830 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 -- 172.21.15.97:0/2927011993 >> 172.21.15.97:0/2927011993 conn(0x7fcf100f0b70 msgr2=0x7fcf100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.269+0000 7fcf176a2700 1 -- 172.21.15.97:0/2927011993 shutdown_connections 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf176a2700 1 -- 172.21.15.97:0/2927011993 wait complete. 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf176a2700 1 Processor -- start 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf176a2700 1 -- start start 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf176a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf176a2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf10106cd0 con 0x7fcf100f5410 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf1543e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf1543e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54372/0 (socket says 172.21.15.97:54372) 2024-02-19T22:19:25.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.270+0000 7fcf1543e700 1 -- 172.21.15.97:0/2592567941 learned_addr learned my addr 172.21.15.97:0/2592567941 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:25.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf1543e700 1 -- 172.21.15.97:0/2592567941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c005040 con 0x7fcf100f5410 2024-02-19T22:19:25.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf1543e700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fcf0c005010 tx=0x7fcf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf0c009320 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf0c013380 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf0c014e50 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf10106ed0 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.271+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf10107360 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.272+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf0c0095f0 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.272+0000 7fcf01ffb700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 0x7fcefc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.272+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf10045ad0 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.272+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf0c040f70 con 0x7fcf100f5410 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.272+0000 7fcf14c3d700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 0x7fcefc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:25.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.273+0000 7fcf14c3d700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 0x7fcefc0323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fcf040096a0 tx=0x7fcf04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:25.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.277+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf0c014420 con 0x7fcf100f5410 2024-02-19T22:19:25.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.482+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf100fb090 con 0x7fcf100f5410 2024-02-19T22:19:25.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.483+0000 7fcf01ffb700 1 -- 172.21.15.97:0/2592567941 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcf0c02c090 con 0x7fcf100f5410 2024-02-19T22:19:25.485 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:25.485 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:25.486 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 msgr2=0x7fcefc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 0x7fcefc0323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fcf040096a0 tx=0x7fcf04006b40).stop 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 msgr2=0x7fcf101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fcf0c005010 tx=0x7fcf0c00b040).stop 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 shutdown_connections 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcefc02ff10 0x7fcefc0323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 --2- 172.21.15.97:0/2592567941 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcf100f5410 0x7fcf101067e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.484+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 >> 172.21.15.97:0/2592567941 conn(0x7fcf100f0b70 msgr2=0x7fcf1005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.485+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 shutdown_connections 2024-02-19T22:19:25.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:25.485+0000 7fcf176a2700 1 -- 172.21.15.97:0/2592567941 wait complete. 2024-02-19T22:19:25.488 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:25.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:23.832455+0000 mgr.smithi045.bvlamo (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:25.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:25.483941+0000 mon.smithi045 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.97:0/2592567941' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:27.198 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:27.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:25.832675+0000 mgr.smithi045.bvlamo (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:29.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 -- 172.21.15.97:0/1584015127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 msgr2=0x7f80580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:29.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 --2- 172.21.15.97:0/1584015127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80580f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f8048009fb0 tx=0x7f8048009b30).stop 2024-02-19T22:19:29.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 -- 172.21.15.97:0/1584015127 shutdown_connections 2024-02-19T22:19:29.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 --2- 172.21.15.97:0/1584015127 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80580f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:29.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 -- 172.21.15.97:0/1584015127 >> 172.21.15.97:0/1584015127 conn(0x7f80580f0b70 msgr2=0x7f80580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:29.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 -- 172.21.15.97:0/1584015127 shutdown_connections 2024-02-19T22:19:29.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.206+0000 7f805fc3f700 1 -- 172.21.15.97:0/1584015127 wait complete. 2024-02-19T22:19:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.207+0000 7f805fc3f700 1 Processor -- start 2024-02-19T22:19:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.207+0000 7f805fc3f700 1 -- start start 2024-02-19T22:19:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.207+0000 7f805fc3f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.207+0000 7f805fc3f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058106ca0 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805d9db700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805d9db700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:49676/0 (socket says 172.21.15.97:49676) 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805d9db700 1 -- 172.21.15.97:0/3480243937 learned_addr learned my addr 172.21.15.97:0/3480243937 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805d9db700 1 -- 172.21.15.97:0/3480243937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048005040 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805d9db700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8048005010 tx=0x7f804800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8048009320 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8058106ea0 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8048013380 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8048014e50 con 0x7f80580f5410 2024-02-19T22:19:29.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.208+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8058107330 con 0x7f80580f5410 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.209+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80480263a0 con 0x7f80580f5410 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.209+0000 7f804e7fc700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 0x7f80440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.209+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8048040fc0 con 0x7f80580f5410 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.209+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8058059100 con 0x7f80580f5410 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.209+0000 7f805d1da700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 0x7f80440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:29.212 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.210+0000 7f805d1da700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 0x7f80440323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:29.216 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.214+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8048012920 con 0x7f80580f5410 2024-02-19T22:19:29.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.419+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8058045ad0 con 0x7f80580f5410 2024-02-19T22:19:29.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.419+0000 7f804e7fc700 1 -- 172.21.15.97:0/3480243937 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f804802b020 con 0x7f80580f5410 2024-02-19T22:19:29.421 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:29.421 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 msgr2=0x7f80440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 0x7f80440323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f80540096a0 tx=0x7f8054006b40).stop 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 msgr2=0x7f80581067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f8048005010 tx=0x7f804800b040).stop 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 shutdown_connections 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f804402ff10 0x7f80440323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 --2- 172.21.15.97:0/3480243937 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80580f5410 0x7f80581067b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:29.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 >> 172.21.15.97:0/3480243937 conn(0x7f80580f0b70 msgr2=0x7f805805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:29.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 shutdown_connections 2024-02-19T22:19:29.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:29.421+0000 7f805fc3f700 1 -- 172.21.15.97:0/3480243937 wait complete. 2024-02-19T22:19:29.424 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:29.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:27.832896+0000 mgr.smithi045.bvlamo (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:29.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:29.420442+0000 mon.smithi045 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.97:0/3480243937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:31.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:31.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:31.871 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:29.833071+0000 mgr.smithi045.bvlamo (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 -- 172.21.15.97:0/4283515973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 msgr2=0x7f664c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 --2- 172.21.15.97:0/4283515973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c0f6930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f6640009fb0 tx=0x7f6640009b30).stop 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 -- 172.21.15.97:0/4283515973 shutdown_connections 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 --2- 172.21.15.97:0/4283515973 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c0f6930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 -- 172.21.15.97:0/4283515973 >> 172.21.15.97:0/4283515973 conn(0x7f664c0f1c70 msgr2=0x7f664c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 -- 172.21.15.97:0/4283515973 shutdown_connections 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 -- 172.21.15.97:0/4283515973 wait complete. 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.922+0000 7f6651831700 1 Processor -- start 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f6651831700 1 -- start start 2024-02-19T22:19:32.925 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f6651831700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.926 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f6651831700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664c106cd0 con 0x7f664c0f6510 2024-02-19T22:19:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f664affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f664affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:49690/0 (socket says 172.21.15.97:49690) 2024-02-19T22:19:32.927 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.923+0000 7f664affd700 1 -- 172.21.15.97:0/4045461826 learned_addr learned my addr 172.21.15.97:0/4045461826 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f664affd700 1 -- 172.21.15.97:0/4045461826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640005040 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f664affd700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6640005010 tx=0x7f664000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6640009320 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6640013380 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c106ed0 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6640014e50 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.924+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c107360 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66400263a0 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f66337fe700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 0x7f66340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6640040f80 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c059100 con 0x7f664c0f6510 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f664a7fc700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 0x7f66340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.928 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.925+0000 7f664a7fc700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 0x7f66340323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f663c0096a0 tx=0x7f663c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:32.930+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6640012920 con 0x7f664c0f6510 2024-02-19T22:19:33.136 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.135+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f664c0fc0a0 con 0x7f664c0f6510 2024-02-19T22:19:33.137 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.136+0000 7f66337fe700 1 -- 172.21.15.97:0/4045461826 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f664002b020 con 0x7f664c0f6510 2024-02-19T22:19:33.138 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:33.138 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 msgr2=0x7f66340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 0x7f66340323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f663c0096a0 tx=0x7f663c006b40).stop 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 msgr2=0x7f664c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f6640005010 tx=0x7f664000b040).stop 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 shutdown_connections 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f663402ff10 0x7f66340323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 --2- 172.21.15.97:0/4045461826 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f664c0f6510 0x7f664c1067e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 >> 172.21.15.97:0/4045461826 conn(0x7f664c0f1c70 msgr2=0x7f664c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:33.139 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 shutdown_connections 2024-02-19T22:19:33.140 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:33.137+0000 7f6651831700 1 -- 172.21.15.97:0/4045461826 wait complete. 2024-02-19T22:19:33.140 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:33.634 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:31.833411+0000 mgr.smithi045.bvlamo (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:33.635 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:33.136553+0000 mon.smithi045 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.97:0/4045461826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:34.799 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:34.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:35.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:33.833814+0000 mgr.smithi045.bvlamo (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:35.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:34.416544+0000 mon.smithi045 (mon.0) 211 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:19:35.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:19:34.417864+0000 mgr.smithi045.bvlamo (mgr.14162) 171 : cephadm [INF] Deploying daemon node-exporter.smithi045 on smithi045 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 -- 172.21.15.97:0/1416760208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 msgr2=0x7f87680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 --2- 172.21.15.97:0/1416760208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f87680f66c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f8758009960 tx=0x7f8758004d10).stop 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 -- 172.21.15.97:0/1416760208 shutdown_connections 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 --2- 172.21.15.97:0/1416760208 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f87680f66c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 -- 172.21.15.97:0/1416760208 >> 172.21.15.97:0/1416760208 conn(0x7f87680f1aa0 msgr2=0x7f87680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 -- 172.21.15.97:0/1416760208 shutdown_connections 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.974+0000 7f876759e700 1 -- 172.21.15.97:0/1416760208 wait complete. 2024-02-19T22:19:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876759e700 1 Processor -- start 2024-02-19T22:19:36.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876759e700 1 -- start start 2024-02-19T22:19:36.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876759e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.978 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876759e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8768106b20 con 0x7f87680f62a0 2024-02-19T22:19:36.979 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:49636/0 (socket says 172.21.15.97:49636) 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.975+0000 7f876659c700 1 -- 172.21.15.97:0/1944139200 learned_addr learned my addr 172.21.15.97:0/1944139200 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f876659c700 1 -- 172.21.15.97:0/1944139200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009640 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f876659c700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8758007b20 tx=0x7f8758004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8758018070 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8768106d20 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8758007e60 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f875801d420 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.976+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87681071b0 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.977+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8758013630 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.977+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8768045ad0 con 0x7f87680f62a0 2024-02-19T22:19:36.980 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.977+0000 7f875effd700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 0x7f8754032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.977+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8758041590 con 0x7f87680f62a0 2024-02-19T22:19:36.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.977+0000 7f8765d9b700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 0x7f8754032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.981 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.978+0000 7f8765d9b700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 0x7f8754032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f87500096a0 tx=0x7f8750006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.984 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:36.982+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8758004810 con 0x7f87680f62a0 2024-02-19T22:19:37.188 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.186+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8768108060 con 0x7f87680f62a0 2024-02-19T22:19:37.189 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.187+0000 7f875effd700 1 -- 172.21.15.97:0/1944139200 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f875801a2d0 con 0x7f87680f62a0 2024-02-19T22:19:37.189 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:37.189 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 msgr2=0x7f8754032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 0x7f8754032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f87500096a0 tx=0x7f8750006b40).stop 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 msgr2=0x7f8768106630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f8758007b20 tx=0x7f8758004d10).stop 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 shutdown_connections 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f875402ff60 0x7f8754032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 --2- 172.21.15.97:0/1944139200 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f87680f62a0 0x7f8768106630 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:37.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 >> 172.21.15.97:0/1944139200 conn(0x7f87680f1aa0 msgr2=0x7f87680f24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 shutdown_connections 2024-02-19T22:19:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:37.189+0000 7f876759e700 1 -- 172.21.15.97:0/1944139200 wait complete. 2024-02-19T22:19:37.192 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:37.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:35.834180+0000 mgr.smithi045.bvlamo (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:37.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:37.188096+0000 mon.smithi045 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.97:0/1944139200' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:39.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:39.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:39.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:37.834478+0000 mgr.smithi045.bvlamo (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- 172.21.15.97:0/529379295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 msgr2=0x7f318c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 --2- 172.21.15.97:0/529379295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c0f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f3180009fb0 tx=0x7f3180009b30).stop 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- 172.21.15.97:0/529379295 shutdown_connections 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 --2- 172.21.15.97:0/529379295 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c0f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- 172.21.15.97:0/529379295 >> 172.21.15.97:0/529379295 conn(0x7f318c0f0b70 msgr2=0x7f318c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- 172.21.15.97:0/529379295 shutdown_connections 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- 172.21.15.97:0/529379295 wait complete. 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 Processor -- start 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.022+0000 7f3193a7e700 1 -- start start 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f3193a7e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f3193a7e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318c060240 con 0x7f318c0f5410 2024-02-19T22:19:41.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f319181a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f319181a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:49654/0 (socket says 172.21.15.97:49654) 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f319181a700 1 -- 172.21.15.97:0/441954971 learned_addr learned my addr 172.21.15.97:0/441954971 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f319181a700 1 -- 172.21.15.97:0/441954971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3180005040 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.023+0000 7f319181a700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3180005010 tx=0x7f318000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3180009320 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c060440 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c109b90 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3180013380 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3180014df0 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31800095f0 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.024+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c045ad0 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.025+0000 7f317e7fc700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 0x7f31780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.025+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3180040eb0 con 0x7f318c0f5410 2024-02-19T22:19:41.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.025+0000 7f3191019700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 0x7f31780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:41.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.025+0000 7f3191019700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 0x7f31780323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f31880096a0 tx=0x7f3188006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:41.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.029+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f318001dbb0 con 0x7f318c0f5410 2024-02-19T22:19:41.240 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.238+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f318c10a3b0 con 0x7f318c0f5410 2024-02-19T22:19:41.241 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.239+0000 7f317e7fc700 1 -- 172.21.15.97:0/441954971 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f318001d9d0 con 0x7f318c0f5410 2024-02-19T22:19:41.241 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:41.241 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 msgr2=0x7f31780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 0x7f31780323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f31880096a0 tx=0x7f3188006b40).stop 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 msgr2=0x7f318c1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f3180005010 tx=0x7f318000b040).stop 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 shutdown_connections 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f317802ff10 0x7f31780323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 --2- 172.21.15.97:0/441954971 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f318c0f5410 0x7f318c1094e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 >> 172.21.15.97:0/441954971 conn(0x7f318c0f0b70 msgr2=0x7f318c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 shutdown_connections 2024-02-19T22:19:41.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:41.241+0000 7f3193a7e700 1 -- 172.21.15.97:0/441954971 wait complete. 2024-02-19T22:19:41.244 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:39.834734+0000 mgr.smithi045.bvlamo (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:41.239800+0000 mon.smithi045 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.97:0/441954971' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:42.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:42.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:43.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:41.835020+0000 mgr.smithi045.bvlamo (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:44.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.939+0000 7fd35b9e4700 1 -- 172.21.15.97:0/3313754451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 msgr2=0x7fd3540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.939+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/3313754451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fd344009fb0 tx=0x7fd344009b30).stop 2024-02-19T22:19:44.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.939+0000 7fd35b9e4700 1 -- 172.21.15.97:0/3313754451 shutdown_connections 2024-02-19T22:19:44.943 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.939+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/3313754451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.939+0000 7fd35b9e4700 1 -- 172.21.15.97:0/3313754451 >> 172.21.15.97:0/3313754451 conn(0x7fd3540f0b70 msgr2=0x7fd3540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 -- 172.21.15.97:0/3313754451 shutdown_connections 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 -- 172.21.15.97:0/3313754451 wait complete. 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 Processor -- start 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 -- start start 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd35b9e4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3540fb960 con 0x7fd3540f5410 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd359780700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd359780700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:49670/0 (socket says 172.21.15.97:49670) 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.940+0000 7fd359780700 1 -- 172.21.15.97:0/1852866401 learned_addr learned my addr 172.21.15.97:0/1852866401 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd359780700 1 -- 172.21.15.97:0/1852866401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd344005040 con 0x7fd3540f5410 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd359780700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fd344005010 tx=0x7fd34400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:44.944 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd344009320 con 0x7fd3540f5410 2024-02-19T22:19:44.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3540fbb00 con 0x7fd3540f5410 2024-02-19T22:19:44.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd344013380 con 0x7fd3540f5410 2024-02-19T22:19:44.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd344014e50 con 0x7fd3540f5410 2024-02-19T22:19:44.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.941+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3540fbf90 con 0x7fd3540f5410 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.942+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3440095f0 con 0x7fd3540f5410 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.942+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd354059100 con 0x7fd3540f5410 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.942+0000 7fd349ffb700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 0x7fd3400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.942+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd344040e70 con 0x7fd3540f5410 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.942+0000 7fd358f7f700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 0x7fd3400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:44.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.943+0000 7fd358f7f700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 0x7fd3400323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fd3500096a0 tx=0x7fd350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:44.949 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:44.947+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd344014420 con 0x7fd3540f5410 2024-02-19T22:19:45.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.152+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd354045ad0 con 0x7fd3540f5410 2024-02-19T22:19:45.154 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.152+0000 7fd349ffb700 1 -- 172.21.15.97:0/1852866401 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd344010230 con 0x7fd3540f5410 2024-02-19T22:19:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:45.155 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:45.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 msgr2=0x7fd3400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:45.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 0x7fd3400323d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fd3500096a0 tx=0x7fd350006b40).stop 2024-02-19T22:19:45.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 msgr2=0x7fd3540fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fd344005010 tx=0x7fd34400b040).stop 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 shutdown_connections 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd34002ff10 0x7fd3400323d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 --2- 172.21.15.97:0/1852866401 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd3540f5410 0x7fd3540fd4e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 >> 172.21.15.97:0/1852866401 conn(0x7fd3540f0b70 msgr2=0x7fd35405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 shutdown_connections 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:45.154+0000 7fd35b9e4700 1 -- 172.21.15.97:0/1852866401 wait complete. 2024-02-19T22:19:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:43.835333+0000 mgr.smithi045.bvlamo (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:45.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:45.153623+0000 mon.smithi045 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.97:0/1852866401' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:46.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:46.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:45.835691+0000 mgr.smithi045.bvlamo (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 -- 172.21.15.97:0/3065719944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 msgr2=0x7fcfb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/3065719944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb40f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fcfa4009fb0 tx=0x7fcfa4009b30).stop 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 -- 172.21.15.97:0/3065719944 shutdown_connections 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/3065719944 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb40f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 -- 172.21.15.97:0/3065719944 >> 172.21.15.97:0/3065719944 conn(0x7fcfb40f0b70 msgr2=0x7fcfb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 -- 172.21.15.97:0/3065719944 shutdown_connections 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.926+0000 7fcfbc72b700 1 -- 172.21.15.97:0/3065719944 wait complete. 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfbc72b700 1 Processor -- start 2024-02-19T22:19:48.930 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfbc72b700 1 -- start start 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfbc72b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfbc72b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4060240 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfba4c7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfba4c7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41462/0 (socket says 172.21.15.97:41462) 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfba4c7700 1 -- 172.21.15.97:0/1672982666 learned_addr learned my addr 172.21.15.97:0/1672982666 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfba4c7700 1 -- 172.21.15.97:0/1672982666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4005040 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.927+0000 7fcfba4c7700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcfa4005010 tx=0x7fcfa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa4009320 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb4060440 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfa4013380 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfa4014e50 con 0x7fcfb40f5410 2024-02-19T22:19:48.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb4109b90 con 0x7fcfb40f5410 2024-02-19T22:19:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.928+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb4045ad0 con 0x7fcfb40f5410 2024-02-19T22:19:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.929+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfa40095f0 con 0x7fcfb40f5410 2024-02-19T22:19:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.929+0000 7fcfaaffd700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.929+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfa4040e90 con 0x7fcfb40f5410 2024-02-19T22:19:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.929+0000 7fcfb9cc6700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.929+0000 7fcfb9cc6700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 0x7fcfa00323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcfb00096a0 tx=0x7fcfb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.935 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:48.933+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfa4014420 con 0x7fcfb40f5410 2024-02-19T22:19:49.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.140+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcfb410a3a0 con 0x7fcfb40f5410 2024-02-19T22:19:49.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.140+0000 7fcfaaffd700 1 -- 172.21.15.97:0/1672982666 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcfa4010230 con 0x7fcfb40f5410 2024-02-19T22:19:49.142 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:49.143 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 msgr2=0x7fcfa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 0x7fcfa00323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fcfb00096a0 tx=0x7fcfb0006b40).stop 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 msgr2=0x7fcfb41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fcfa4005010 tx=0x7fcfa400b040).stop 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 shutdown_connections 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcfa002ff10 0x7fcfa00323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 --2- 172.21.15.97:0/1672982666 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcfb40f5410 0x7fcfb41094e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 >> 172.21.15.97:0/1672982666 conn(0x7fcfb40f0b70 msgr2=0x7fcfb40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 shutdown_connections 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:49.142+0000 7fcfbc72b700 1 -- 172.21.15.97:0/1672982666 wait complete. 2024-02-19T22:19:49.145 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:49.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:47.836083+0000 mgr.smithi045.bvlamo (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:49.141466+0000 mon.smithi045 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.97:0/1672982666' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:50.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:50.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:49.836466+0000 mgr.smithi045.bvlamo (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 -- 172.21.15.97:0/1131415002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 msgr2=0x7f634c0f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 --2- 172.21.15.97:0/1131415002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c0f5810 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f6340009fb0 tx=0x7f6340009b30).stop 2024-02-19T22:19:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 -- 172.21.15.97:0/1131415002 shutdown_connections 2024-02-19T22:19:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 --2- 172.21.15.97:0/1131415002 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c0f5810 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.956 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 -- 172.21.15.97:0/1131415002 >> 172.21.15.97:0/1131415002 conn(0x7f634c0f0b70 msgr2=0x7f634c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:52.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 -- 172.21.15.97:0/1131415002 shutdown_connections 2024-02-19T22:19:52.957 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.954+0000 7f635182f700 1 -- 172.21.15.97:0/1131415002 wait complete. 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f635182f700 1 Processor -- start 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f635182f700 1 -- start start 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f635182f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f635182f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c107270 con 0x7f634c0f53f0 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f634affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.958 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f634affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41488/0 (socket says 172.21.15.97:41488) 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.955+0000 7f634affd700 1 -- 172.21.15.97:0/2719002837 learned_addr learned my addr 172.21.15.97:0/2719002837 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f634affd700 1 -- 172.21.15.97:0/2719002837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6340005040 con 0x7f634c0f53f0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f634affd700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6340005010 tx=0x7f634000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6340009320 con 0x7f634c0f53f0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c060280 con 0x7f634c0f53f0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6340013380 con 0x7f634c0f53f0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6340014e50 con 0x7f634c0f53f0 2024-02-19T22:19:52.959 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.956+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c10bd70 con 0x7f634c0f53f0 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.957+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6340014420 con 0x7f634c0f53f0 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.957+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c045ad0 con 0x7f634c0f53f0 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.957+0000 7f63337fe700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 0x7f63340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.957+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6340040e30 con 0x7f634c0f53f0 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.957+0000 7f634a7fc700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 0x7f63340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.958+0000 7f634a7fc700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 0x7f63340323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f633c0096a0 tx=0x7f633c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.964 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:52.962+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63400095f0 con 0x7f634c0f53f0 2024-02-19T22:19:53.170 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.168+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f634c10c590 con 0x7f634c0f53f0 2024-02-19T22:19:53.171 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.169+0000 7f63337fe700 1 -- 172.21.15.97:0/2719002837 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f634001db70 con 0x7f634c0f53f0 2024-02-19T22:19:53.171 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:53.171 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:53.172 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 msgr2=0x7f63340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 0x7f63340323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f633c0096a0 tx=0x7f633c006b40).stop 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 msgr2=0x7f634c10b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f6340005010 tx=0x7f634000b040).stop 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 shutdown_connections 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f633402ff10 0x7f63340323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 --2- 172.21.15.97:0/2719002837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f634c0f53f0 0x7f634c10b880 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 >> 172.21.15.97:0/2719002837 conn(0x7f634c0f0b70 msgr2=0x7f634c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 shutdown_connections 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:53.170+0000 7f635182f700 1 -- 172.21.15.97:0/2719002837 wait complete. 2024-02-19T22:19:53.173 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:51.836825+0000 mgr.smithi045.bvlamo (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:53.169867+0000 mon.smithi045 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.97:0/2719002837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:54.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:54.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:53.837167+0000 mgr.smithi045.bvlamo (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:57.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 -- 172.21.15.97:0/1892614696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 msgr2=0x7fdc4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:57.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1892614696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c0f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fdc3c009fb0 tx=0x7fdc3c009b30).stop 2024-02-19T22:19:57.063 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 -- 172.21.15.97:0/1892614696 shutdown_connections 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1892614696 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c0f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 -- 172.21.15.97:0/1892614696 >> 172.21.15.97:0/1892614696 conn(0x7fdc4c0f0b70 msgr2=0x7fdc4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 -- 172.21.15.97:0/1892614696 shutdown_connections 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 -- 172.21.15.97:0/1892614696 wait complete. 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.061+0000 7fdc5292f700 1 Processor -- start 2024-02-19T22:19:57.064 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc5292f700 1 -- start start 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc5292f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc5292f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c105bd0 con 0x7fdc4c0f5410 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc4bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc4bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34892/0 (socket says 172.21.15.97:34892) 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc4bfff700 1 -- 172.21.15.97:0/1826384004 learned_addr learned my addr 172.21.15.97:0/1826384004 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.062+0000 7fdc4bfff700 1 -- 172.21.15.97:0/1826384004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c005040 con 0x7fdc4c0f5410 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc4bfff700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fdc3c005010 tx=0x7fdc3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c009320 con 0x7fdc4c0f5410 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c105dd0 con 0x7fdc4c0f5410 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc3c013380 con 0x7fdc4c0f5410 2024-02-19T22:19:57.066 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3c014e50 con 0x7fdc4c0f5410 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.063+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c106260 con 0x7fdc4c0f5410 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc3c014420 con 0x7fdc4c0f5410 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c045ad0 con 0x7fdc4c0f5410 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc48ff9700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc3c042c20 con 0x7fdc4c0f5410 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc4b7fe700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:57.067 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.064+0000 7fdc4b7fe700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 0x7fdc340323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:57.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.069+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc3c0095f0 con 0x7fdc4c0f5410 2024-02-19T22:19:57.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.278+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc4c107110 con 0x7fdc4c0f5410 2024-02-19T22:19:57.280 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.278+0000 7fdc48ff9700 1 -- 172.21.15.97:0/1826384004 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc3c01fb70 con 0x7fdc4c0f5410 2024-02-19T22:19:57.280 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:19:57.281 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 msgr2=0x7fdc340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 0x7fdc340323d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fdc400096a0 tx=0x7fdc40006b40).stop 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 msgr2=0x7fdc4c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fdc3c005010 tx=0x7fdc3c00b040).stop 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 shutdown_connections 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fdc3402ff10 0x7fdc340323d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:57.282 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 --2- 172.21.15.97:0/1826384004 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fdc4c0f5410 0x7fdc4c1056e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:57.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 >> 172.21.15.97:0/1826384004 conn(0x7fdc4c0f0b70 msgr2=0x7fdc4c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:57.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 shutdown_connections 2024-02-19T22:19:57.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:19:57.280+0000 7fdc5292f700 1 -- 172.21.15.97:0/1826384004 wait complete. 2024-02-19T22:19:57.283 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:19:57.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:55.837445+0000 mgr.smithi045.bvlamo (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:19:57.279627+0000 mon.smithi045 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.97:0/1826384004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:59.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:59.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:19:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:57.837785+0000 mgr.smithi045.bvlamo (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:00.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 -- 172.21.15.97:0/2097741812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 msgr2=0x7f56dc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 --2- 172.21.15.97:0/2097741812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc0f66c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f56d0009fb0 tx=0x7f56d0009b30).stop 2024-02-19T22:20:00.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 -- 172.21.15.97:0/2097741812 shutdown_connections 2024-02-19T22:20:00.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 --2- 172.21.15.97:0/2097741812 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc0f66c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 -- 172.21.15.97:0/2097741812 >> 172.21.15.97:0/2097741812 conn(0x7f56dc0f1aa0 msgr2=0x7f56dc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 -- 172.21.15.97:0/2097741812 shutdown_connections 2024-02-19T22:20:00.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.996+0000 7f56dbfff700 1 -- 172.21.15.97:0/2097741812 wait complete. 2024-02-19T22:20:01.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56dbfff700 1 Processor -- start 2024-02-19T22:20:01.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56dbfff700 1 -- start start 2024-02-19T22:20:01.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56dbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56dbfff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56dc106ac0 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56daffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.997+0000 7f56daffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34910/0 (socket says 172.21.15.97:34910) 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56daffd700 1 -- 172.21.15.97:0/487569368 learned_addr learned my addr 172.21.15.97:0/487569368 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56daffd700 1 -- 172.21.15.97:0/487569368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0005040 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56daffd700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f56d00089f0 tx=0x7f56d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d0009320 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56dc106cc0 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56d0013380 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56d0014e20 con 0x7f56dc0f62a0 2024-02-19T22:20:01.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.998+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56dc107150 con 0x7f56dc0f62a0 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.999+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56d00095f0 con 0x7f56dc0f62a0 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.999+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc045ad0 con 0x7f56dc0f62a0 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.999+0000 7f56c37fe700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 0x7f56c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.999+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56d0040e90 con 0x7f56dc0f62a0 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:00.999+0000 7f56da7fc700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 0x7f56c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:01.002 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.000+0000 7f56da7fc700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 0x7f56c40323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f56cc0096a0 tx=0x7f56cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:01.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.004+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56d0014420 con 0x7f56dc0f62a0 2024-02-19T22:20:01.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.208+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56dc108000 con 0x7f56dc0f62a0 2024-02-19T22:20:01.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.209+0000 7f56c37fe700 1 -- 172.21.15.97:0/487569368 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56d001d980 con 0x7f56dc0f62a0 2024-02-19T22:20:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:01.212 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 msgr2=0x7f56c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 0x7f56c40323d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f56cc0096a0 tx=0x7f56cc006b40).stop 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 msgr2=0x7f56dc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f56d00089f0 tx=0x7f56d000b040).stop 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 shutdown_connections 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56c402ff10 0x7f56c40323d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 --2- 172.21.15.97:0/487569368 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56dc0f62a0 0x7f56dc1065d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 >> 172.21.15.97:0/487569368 conn(0x7f56dc0f1aa0 msgr2=0x7f56dc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:01.213 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 shutdown_connections 2024-02-19T22:20:01.214 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:01.211+0000 7f56dbfff700 1 -- 172.21.15.97:0/487569368 wait complete. 2024-02-19T22:20:01.214 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:19:59.838166+0000 mgr.smithi045.bvlamo (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:01.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:01.210486+0000 mon.smithi045 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.97:0/487569368' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:02.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:02.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:01.838537+0000 mgr.smithi045.bvlamo (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:03.935209+0000 mon.smithi045 (mon.0) 219 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:20:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:03.936067+0000 mon.smithi045 (mon.0) 220 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:20:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.941+0000 7f4aab116700 1 -- 172.21.15.97:0/2300117431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f47c0 msgr2=0x7f4aa40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.941+0000 7f4aab116700 1 --2- 172.21.15.97:0/2300117431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f47c0 0x7f4aa40f4be0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f4a98009fb0 tx=0x7f4a98009b30).stop 2024-02-19T22:20:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.941+0000 7f4aab116700 1 -- 172.21.15.97:0/2300117431 shutdown_connections 2024-02-19T22:20:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.941+0000 7f4aab116700 1 --2- 172.21.15.97:0/2300117431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f47c0 0x7f4aa40f4be0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.941+0000 7f4aab116700 1 -- 172.21.15.97:0/2300117431 >> 172.21.15.97:0/2300117431 conn(0x7f4aa40f02e0 msgr2=0x7f4aa40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 -- 172.21.15.97:0/2300117431 shutdown_connections 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 -- 172.21.15.97:0/2300117431 wait complete. 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 Processor -- start 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 -- start start 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.942+0000 7f4aab116700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aa40f7750 con 0x7f4aa40f7330 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa8eb2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa8eb2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34928/0 (socket says 172.21.15.97:34928) 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa8eb2700 1 -- 172.21.15.97:0/484275958 learned_addr learned my addr 172.21.15.97:0/484275958 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa8eb2700 1 -- 172.21.15.97:0/484275958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98005040 con 0x7f4aa40f7330 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa8eb2700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4aa405f2f0 tx=0x7f4a98009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a98013380 con 0x7f4aa40f7330 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a98013790 con 0x7f4aa40f7330 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aa40f5ce0 con 0x7f4aa40f7330 2024-02-19T22:20:04.946 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a9801ee70 con 0x7f4aa40f7330 2024-02-19T22:20:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.943+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa40f61d0 con 0x7f4aa40f7330 2024-02-19T22:20:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.944+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a98025060 con 0x7f4aa40f7330 2024-02-19T22:20:04.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.944+0000 7f4aa17fa700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 0x7f4a94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.944+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a98040db0 con 0x7f4aa40f7330 2024-02-19T22:20:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.944+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aa4045ad0 con 0x7f4aa40f7330 2024-02-19T22:20:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.944+0000 7f4aa3fff700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 0x7f4a94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.948 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.945+0000 7f4aa3fff700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 0x7f4a94032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4a900096a0 tx=0x7f4a90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.951 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:04.949+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a98028440 con 0x7f4aa40f7330 2024-02-19T22:20:05.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.153+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4aa410c9d0 con 0x7f4aa40f7330 2024-02-19T22:20:05.156 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.154+0000 7f4aa17fa700 1 -- 172.21.15.97:0/484275958 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a9801d3a0 con 0x7f4aa40f7330 2024-02-19T22:20:05.156 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:05.156 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 msgr2=0x7f4a94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 0x7f4a94032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4a900096a0 tx=0x7f4a90006b40).stop 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 msgr2=0x7f4aa40f57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f4aa405f2f0 tx=0x7f4a98009b30).stop 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 shutdown_connections 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4a9402ff60 0x7f4a94032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 --2- 172.21.15.97:0/484275958 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4aa40f7330 0x7f4aa40f57f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 >> 172.21.15.97:0/484275958 conn(0x7f4aa40f02e0 msgr2=0x7f4aa40f0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 shutdown_connections 2024-02-19T22:20:05.158 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:05.156+0000 7f4aab116700 1 -- 172.21.15.97:0/484275958 wait complete. 2024-02-19T22:20:05.159 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:05.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:03.838948+0000 mgr.smithi045.bvlamo (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:05.155302+0000 mon.smithi045 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.97:0/484275958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:06.800 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:06.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:07.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:05.839315+0000 mgr.smithi045.bvlamo (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:08.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/2182105213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 msgr2=0x7f7bf00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/2182105213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf00f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f7bec009fb0 tx=0x7f7bec009b30).stop 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/2182105213 shutdown_connections 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/2182105213 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf00f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/2182105213 >> 172.21.15.97:0/2182105213 conn(0x7f7bf00f0b70 msgr2=0x7f7bf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/2182105213 shutdown_connections 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/2182105213 wait complete. 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 Processor -- start 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.732+0000 7f7bf7bd3700 1 -- start start 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf7bd3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:08.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf7bd3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf0105bd0 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf596f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf596f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35872/0 (socket says 172.21.15.97:35872) 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf596f700 1 -- 172.21.15.97:0/425731721 learned_addr learned my addr 172.21.15.97:0/425731721 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf596f700 1 -- 172.21.15.97:0/425731721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec005040 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7bf596f700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f7bec0089f0 tx=0x7f7bec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.733+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec009320 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf0105dd0 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bec013380 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bec020d80 con 0x7f7bf00f5410 2024-02-19T22:20:08.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf0106260 con 0x7f7bf00f5410 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bec014420 con 0x7f7bf00f5410 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.734+0000 7f7be67fc700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.735+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bec042db0 con 0x7f7bf00f5410 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.735+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf0059100 con 0x7f7bf00f5410 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.735+0000 7f7bf516e700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:08.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.735+0000 7f7bf516e700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 0x7f7bdc032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7be00096a0 tx=0x7f7be0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:08.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.740+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bec01e500 con 0x7f7bf00f5410 2024-02-19T22:20:08.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.942+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bf00fafa0 con 0x7f7bf00f5410 2024-02-19T22:20:08.945 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.943+0000 7f7be67fc700 1 -- 172.21.15.97:0/425731721 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7bec01e320 con 0x7f7bf00f5410 2024-02-19T22:20:08.945 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:08.945 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 msgr2=0x7f7bdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 0x7f7bdc032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f7be00096a0 tx=0x7f7be0006b40).stop 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 msgr2=0x7f7bf01056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f7bec0089f0 tx=0x7f7bec00b040).stop 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 shutdown_connections 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7bdc02ff60 0x7f7bdc032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 --2- 172.21.15.97:0/425731721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7bf00f5410 0x7f7bf01056e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 >> 172.21.15.97:0/425731721 conn(0x7f7bf00f0b70 msgr2=0x7f7bf00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 shutdown_connections 2024-02-19T22:20:08.947 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:08.945+0000 7f7bf7bd3700 1 -- 172.21.15.97:0/425731721 wait complete. 2024-02-19T22:20:08.948 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:07.839700+0000 mgr.smithi045.bvlamo (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:09.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:08.944273+0000 mon.smithi045 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.97:0/425731721' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:10.612 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:10.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:09.840114+0000 mgr.smithi045.bvlamo (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:12.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 -- 172.21.15.97:0/2138908561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 msgr2=0x7fe11c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 --2- 172.21.15.97:0/2138908561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c0f90f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fe10c009fb0 tx=0x7fe10c009b30).stop 2024-02-19T22:20:12.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 -- 172.21.15.97:0/2138908561 shutdown_connections 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 --2- 172.21.15.97:0/2138908561 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c0f90f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 -- 172.21.15.97:0/2138908561 >> 172.21.15.97:0/2138908561 conn(0x7fe11c0f0b70 msgr2=0x7fe11c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 -- 172.21.15.97:0/2138908561 shutdown_connections 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.675+0000 7fe1229a2700 1 -- 172.21.15.97:0/2138908561 wait complete. 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe1229a2700 1 Processor -- start 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe1229a2700 1 -- start start 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe1229a2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe1229a2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c106c70 con 0x7fe11c060280 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe11bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe11bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35890/0 (socket says 172.21.15.97:35890) 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe11bfff700 1 -- 172.21.15.97:0/1865968383 learned_addr learned my addr 172.21.15.97:0/1865968383 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.676+0000 7fe11bfff700 1 -- 172.21.15.97:0/1865968383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c005040 con 0x7fe11c060280 2024-02-19T22:20:12.693 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe11bfff700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fe10c0089f0 tx=0x7fe10c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c009320 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe10c013380 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c106e70 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe10c014e20 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.677+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c107300 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe10c0095f0 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c045ad0 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe118ff9700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 0x7fe1040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe10c040e90 con 0x7fe11c060280 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe11b7fe700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 0x7fe1040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.678+0000 7fe11b7fe700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 0x7fe1040323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe1100096a0 tx=0x7fe110006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:12.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.683+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe10c014420 con 0x7fe11c060280 2024-02-19T22:20:12.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.889+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe11c1081b0 con 0x7fe11c060280 2024-02-19T22:20:12.892 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.890+0000 7fe118ff9700 1 -- 172.21.15.97:0/1865968383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe10c01d980 con 0x7fe11c060280 2024-02-19T22:20:12.892 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:12.892 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 msgr2=0x7fe1040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 0x7fe1040323d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fe1100096a0 tx=0x7fe110006b40).stop 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 msgr2=0x7fe11c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fe10c0089f0 tx=0x7fe10c00b040).stop 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 shutdown_connections 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe10402ff10 0x7fe1040323d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 --2- 172.21.15.97:0/1865968383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe11c060280 0x7fe11c106780 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 >> 172.21.15.97:0/1865968383 conn(0x7fe11c0f0b70 msgr2=0x7fe11c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 shutdown_connections 2024-02-19T22:20:12.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:12.892+0000 7fe1229a2700 1 -- 172.21.15.97:0/1865968383 wait complete. 2024-02-19T22:20:12.895 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:13.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:11.840475+0000 mgr.smithi045.bvlamo (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:12.891304+0000 mon.smithi045 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.97:0/1865968383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:14.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:14.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:13.840873+0000 mgr.smithi045.bvlamo (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 -- 172.21.15.97:0/1622047814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 msgr2=0x7f7a200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 --2- 172.21.15.97:0/1622047814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a200f6930 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f7a10009fb0 tx=0x7f7a10009b30).stop 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 -- 172.21.15.97:0/1622047814 shutdown_connections 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 --2- 172.21.15.97:0/1622047814 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a200f6930 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 -- 172.21.15.97:0/1622047814 >> 172.21.15.97:0/1622047814 conn(0x7f7a200f1c70 msgr2=0x7f7a200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 -- 172.21.15.97:0/1622047814 shutdown_connections 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.650+0000 7f7a279a0700 1 -- 172.21.15.97:0/1622047814 wait complete. 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a279a0700 1 Processor -- start 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a279a0700 1 -- start start 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a279a0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a279a0700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20069e20 con 0x7f7a200f6510 2024-02-19T22:20:16.654 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a2573c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a2573c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:36730/0 (socket says 172.21.15.97:36730) 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a2573c700 1 -- 172.21.15.97:0/2913033188 learned_addr learned my addr 172.21.15.97:0/2913033188 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.651+0000 7f7a2573c700 1 -- 172.21.15.97:0/2913033188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10005040 con 0x7f7a200f6510 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a2573c700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f7a10005010 tx=0x7f7a1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a10009320 con 0x7f7a200f6510 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7a10013380 con 0x7f7a200f6510 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a20069fc0 con 0x7f7a200f6510 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a10014e50 con 0x7f7a200f6510 2024-02-19T22:20:16.655 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.652+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2006a450 con 0x7f7a200f6510 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.653+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a10014420 con 0x7f7a200f6510 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.653+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20059100 con 0x7f7a200f6510 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.653+0000 7f7a15ffb700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 0x7f7a0c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.653+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a10040f70 con 0x7f7a200f6510 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.653+0000 7f7a24f3b700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 0x7f7a0c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:16.656 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.654+0000 7f7a24f3b700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 0x7f7a0c0323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7a1c0096a0 tx=0x7f7a1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:16.659 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.658+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a100095f0 con 0x7f7a200f6510 2024-02-19T22:20:16.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.860+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a20045ad0 con 0x7f7a200f6510 2024-02-19T22:20:16.863 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.861+0000 7f7a15ffb700 1 -- 172.21.15.97:0/2913033188 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7a1001db70 con 0x7f7a200f6510 2024-02-19T22:20:16.863 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:16.863 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.862+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 msgr2=0x7f7a0c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.862+0000 7f7a279a0700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 0x7f7a0c0323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f7a1c0096a0 tx=0x7f7a1c006b40).stop 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.862+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 msgr2=0x7f7a2006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.862+0000 7f7a279a0700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f7a10005010 tx=0x7f7a1000b040).stop 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 shutdown_connections 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7a0c02ff10 0x7f7a0c0323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 --2- 172.21.15.97:0/2913033188 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7a200f6510 0x7f7a2006b9a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 >> 172.21.15.97:0/2913033188 conn(0x7f7a200f1c70 msgr2=0x7f7a200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 shutdown_connections 2024-02-19T22:20:16.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:16.863+0000 7f7a279a0700 1 -- 172.21.15.97:0/2913033188 wait complete. 2024-02-19T22:20:16.866 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:15.841285+0000 mgr.smithi045.bvlamo (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:16.862078+0000 mon.smithi045 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.97:0/2913033188' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:18.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:18.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:19.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:17.841622+0000 mgr.smithi045.bvlamo (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 -- 172.21.15.97:0/654419451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 msgr2=0x7f92cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 --2- 172.21.15.97:0/654419451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc0f55c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f92bc009fb0 tx=0x7f92bc009b30).stop 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 -- 172.21.15.97:0/654419451 shutdown_connections 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 --2- 172.21.15.97:0/654419451 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc0f55c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 -- 172.21.15.97:0/654419451 >> 172.21.15.97:0/654419451 conn(0x7f92cc0f09a0 msgr2=0x7f92cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:20.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 -- 172.21.15.97:0/654419451 shutdown_connections 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 -- 172.21.15.97:0/654419451 wait complete. 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.474+0000 7f92d15ef700 1 Processor -- start 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92d15ef700 1 -- start start 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92d15ef700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92d15ef700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc1059c0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92cbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92cbfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:36756/0 (socket says 172.21.15.97:36756) 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92cbfff700 1 -- 172.21.15.97:0/3414326837 learned_addr learned my addr 172.21.15.97:0/3414326837 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.475+0000 7f92cbfff700 1 -- 172.21.15.97:0/3414326837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc005040 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92cbfff700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f92bc0089f0 tx=0x7f92bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92bc009320 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc105bc0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f92bc013380 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92bc014e20 con 0x7f92cc0f51a0 2024-02-19T22:20:20.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.476+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc106050 con 0x7f92cc0f51a0 2024-02-19T22:20:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.477+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92bc014420 con 0x7f92cc0f51a0 2024-02-19T22:20:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.477+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc045ad0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.477+0000 7f92c8ff9700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 0x7f92b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.477+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92bc040ed0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.477+0000 7f92cb7fe700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 0x7f92b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:20.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.478+0000 7f92cb7fe700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 0x7f92b40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f92c00096a0 tx=0x7f92c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:20.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.482+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92bc014790 con 0x7f92cc0f51a0 2024-02-19T22:20:20.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.685+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92cc106ea0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.719 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.686+0000 7f92c8ff9700 1 -- 172.21.15.97:0/3414326837 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f92bc01d7f0 con 0x7f92cc0f51a0 2024-02-19T22:20:20.719 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.687+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 msgr2=0x7f92b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.687+0000 7f92d15ef700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 0x7f92b40323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f92c00096a0 tx=0x7f92c0006b40).stop 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.687+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 msgr2=0x7f92cc1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f92bc0089f0 tx=0x7f92bc00b040).stop 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 shutdown_connections 2024-02-19T22:20:20.720 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f92b402ff10 0x7f92b40323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 --2- 172.21.15.97:0/3414326837 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f92cc0f51a0 0x7f92cc1054d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 >> 172.21.15.97:0/3414326837 conn(0x7f92cc0f09a0 msgr2=0x7f92cc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 shutdown_connections 2024-02-19T22:20:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:20.688+0000 7f92d15ef700 1 -- 172.21.15.97:0/3414326837 wait complete. 2024-02-19T22:20:20.721 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:21.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:19.841978+0000 mgr.smithi045.bvlamo (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:21.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:20.687185+0000 mon.smithi045 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.97:0/3414326837' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:22.374 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:22.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:21.842327+0000 mgr.smithi045.bvlamo (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 -- 172.21.15.97:0/1697660238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 msgr2=0x7fea880f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 --2- 172.21.15.97:0/1697660238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea880f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fea78009fb0 tx=0x7fea78009b30).stop 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 -- 172.21.15.97:0/1697660238 shutdown_connections 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 --2- 172.21.15.97:0/1697660238 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea880f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 -- 172.21.15.97:0/1697660238 >> 172.21.15.97:0/1697660238 conn(0x7fea880f09a0 msgr2=0x7fea880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:24.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 -- 172.21.15.97:0/1697660238 shutdown_connections 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.298+0000 7fea87fff700 1 -- 172.21.15.97:0/1697660238 wait complete. 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.299+0000 7fea87fff700 1 Processor -- start 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.299+0000 7fea87fff700 1 -- start start 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.299+0000 7fea87fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.299+0000 7fea87fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea88106ac0 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea86ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea86ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:36774/0 (socket says 172.21.15.97:36774) 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea86ffd700 1 -- 172.21.15.97:0/3913680582 learned_addr learned my addr 172.21.15.97:0/3913680582 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea86ffd700 1 -- 172.21.15.97:0/3913680582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea78005040 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea86ffd700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fea780089f0 tx=0x7fea7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea78009320 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea88106cc0 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea88107150 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea78013380 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.300+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea78014e20 con 0x7fea880f51a0 2024-02-19T22:20:24.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.301+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea780095f0 con 0x7fea880f51a0 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.301+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea88045ad0 con 0x7fea880f51a0 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.301+0000 7fea7f7fe700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 0x7fea740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.301+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea78040f50 con 0x7fea880f51a0 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.301+0000 7fea867fc700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 0x7fea740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.302+0000 7fea867fc700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 0x7fea740323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fea700096a0 tx=0x7fea70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:24.328 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.306+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea78014420 con 0x7fea880f51a0 2024-02-19T22:20:24.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.511+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea88108770 con 0x7fea880f51a0 2024-02-19T22:20:24.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.512+0000 7fea7f7fe700 1 -- 172.21.15.97:0/3913680582 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea78010230 con 0x7fea880f51a0 2024-02-19T22:20:24.514 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:24.514 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 msgr2=0x7fea740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 0x7fea740323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fea700096a0 tx=0x7fea70006b40).stop 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 msgr2=0x7fea881065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7fea780089f0 tx=0x7fea7800b040).stop 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 shutdown_connections 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fea7402ff10 0x7fea740323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 --2- 172.21.15.97:0/3913680582 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fea880f51a0 0x7fea881065d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 >> 172.21.15.97:0/3913680582 conn(0x7fea880f09a0 msgr2=0x7fea8805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 shutdown_connections 2024-02-19T22:20:24.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:24.514+0000 7fea87fff700 1 -- 172.21.15.97:0/3913680582 wait complete. 2024-02-19T22:20:24.517 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:24.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:24.513214+0000 mon.smithi045 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.97:0/3913680582' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:25.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:23.842594+0000 mgr.smithi045.bvlamo (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:26.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:26.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:25.842954+0000 mgr.smithi045.bvlamo (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:28.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 -- 172.21.15.97:0/3933245453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 msgr2=0x7fee3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:28.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 --2- 172.21.15.97:0/3933245453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fee34009fb0 tx=0x7fee34009b30).stop 2024-02-19T22:20:28.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 -- 172.21.15.97:0/3933245453 shutdown_connections 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 --2- 172.21.15.97:0/3933245453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 -- 172.21.15.97:0/3933245453 >> 172.21.15.97:0/3933245453 conn(0x7fee3c0f0b70 msgr2=0x7fee3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 -- 172.21.15.97:0/3933245453 shutdown_connections 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.305+0000 7fee43e16700 1 -- 172.21.15.97:0/3933245453 wait complete. 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee43e16700 1 Processor -- start 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee43e16700 1 -- start start 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee43e16700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee43e16700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee3c0fb960 con 0x7fee3c0f5410 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee41bb2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee41bb2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:51990/0 (socket says 172.21.15.97:51990) 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee41bb2700 1 -- 172.21.15.97:0/1495785212 learned_addr learned my addr 172.21.15.97:0/1495785212 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.306+0000 7fee41bb2700 1 -- 172.21.15.97:0/1495785212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee34005040 con 0x7fee3c0f5410 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee41bb2700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fee34005010 tx=0x7fee3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:28.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee34009320 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c0fbb00 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee34013380 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee34014e50 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.307+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c0fbf90 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee340095f0 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3c059100 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee3a7fc700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 0x7fee240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee34040f80 con 0x7fee3c0f5410 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee413b1700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 0x7fee240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.308+0000 7fee413b1700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 0x7fee240323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fee300089e0 tx=0x7fee30006ec0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:28.349 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.313+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee34014420 con 0x7fee3c0f5410 2024-02-19T22:20:28.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.523+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee3c0fafa0 con 0x7fee3c0f5410 2024-02-19T22:20:28.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.524+0000 7fee3a7fc700 1 -- 172.21.15.97:0/1495785212 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee34010230 con 0x7fee3c0f5410 2024-02-19T22:20:28.526 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:28.526 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.525+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 msgr2=0x7fee240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.525+0000 7fee43e16700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 0x7fee240323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fee300089e0 tx=0x7fee30006ec0).stop 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.525+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 msgr2=0x7fee3c0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.525+0000 7fee43e16700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fee34005010 tx=0x7fee3400b040).stop 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 shutdown_connections 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee2402ff10 0x7fee240323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 --2- 172.21.15.97:0/1495785212 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee3c0f5410 0x7fee3c0fd4e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 >> 172.21.15.97:0/1495785212 conn(0x7fee3c0f0b70 msgr2=0x7fee3c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 shutdown_connections 2024-02-19T22:20:28.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:28.526+0000 7fee43e16700 1 -- 172.21.15.97:0/1495785212 wait complete. 2024-02-19T22:20:28.529 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:28.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:28.524859+0000 mon.smithi045 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.97:0/1495785212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:29.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:27.843162+0000 mgr.smithi045.bvlamo (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:30.332 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:30.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:31.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:29.843429+0000 mgr.smithi045.bvlamo (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:32.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.274+0000 7f7f2759e700 1 -- 172.21.15.97:0/23966336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 msgr2=0x7f7f280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:32.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.274+0000 7f7f2759e700 1 --2- 172.21.15.97:0/23966336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f280f66c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f7f1c009fb0 tx=0x7f7f1c009b30).stop 2024-02-19T22:20:32.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- 172.21.15.97:0/23966336 shutdown_connections 2024-02-19T22:20:32.283 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 --2- 172.21.15.97:0/23966336 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f280f66c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- 172.21.15.97:0/23966336 >> 172.21.15.97:0/23966336 conn(0x7f7f280f1aa0 msgr2=0x7f7f280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- 172.21.15.97:0/23966336 shutdown_connections 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- 172.21.15.97:0/23966336 wait complete. 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 Processor -- start 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- start start 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2759e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28069e20 con 0x7f7f280f62a0 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52006/0 (socket says 172.21.15.97:52006) 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.275+0000 7f7f2659c700 1 -- 172.21.15.97:0/1649908383 learned_addr learned my addr 172.21.15.97:0/1649908383 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f2659c700 1 -- 172.21.15.97:0/1649908383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c005040 con 0x7f7f280f62a0 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f2659c700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7f1c009830 tx=0x7f7f1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c009320 con 0x7f7f280f62a0 2024-02-19T22:20:32.284 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f28069fc0 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f1c013380 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f1c014420 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.276+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2806a450 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.277+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f1c01e420 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.277+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28059100 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.277+0000 7f7f16ffd700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.277+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f1c040e20 con 0x7f7f280f62a0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.277+0000 7f7f25d9b700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:32.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.278+0000 7f7f25d9b700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 0x7f7f10032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7f180096a0 tx=0x7f7f18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:32.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.282+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f1c01c930 con 0x7f7f280f62a0 2024-02-19T22:20:32.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.488+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f280fbf80 con 0x7f7f280f62a0 2024-02-19T22:20:32.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.488+0000 7f7f16ffd700 1 -- 172.21.15.97:0/1649908383 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7f1c02c2d0 con 0x7f7f280f62a0 2024-02-19T22:20:32.491 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:32.491 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:32.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 msgr2=0x7f7f10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:32.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 0x7f7f10032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f7f180096a0 tx=0x7f7f18006b40).stop 2024-02-19T22:20:32.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 msgr2=0x7f7f2806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f7f1c009830 tx=0x7f7f1c00b040).stop 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 shutdown_connections 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7f1002ff60 0x7f7f10032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 --2- 172.21.15.97:0/1649908383 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7f280f62a0 0x7f7f2806b9a0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 >> 172.21.15.97:0/1649908383 conn(0x7f7f280f1aa0 msgr2=0x7f7f280f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 shutdown_connections 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:32.490+0000 7f7f2759e700 1 -- 172.21.15.97:0/1649908383 wait complete. 2024-02-19T22:20:32.493 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:32.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:32.489693+0000 mon.smithi045 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.97:0/1649908383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:33.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:31.843770+0000 mgr.smithi045.bvlamo (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:34.264 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:34.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:35.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:33.844097+0000 mgr.smithi045.bvlamo (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 -- 172.21.15.97:0/2444294372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 msgr2=0x7f6fdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2444294372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc0f5830 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f6fd8009fb0 tx=0x7f6fd8009b30).stop 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 -- 172.21.15.97:0/2444294372 shutdown_connections 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2444294372 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc0f5830 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 -- 172.21.15.97:0/2444294372 >> 172.21.15.97:0/2444294372 conn(0x7f6fdc0f0b70 msgr2=0x7f6fdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 -- 172.21.15.97:0/2444294372 shutdown_connections 2024-02-19T22:20:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.249+0000 7f6fe368a700 1 -- 172.21.15.97:0/2444294372 wait complete. 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe368a700 1 Processor -- start 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe368a700 1 -- start start 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe368a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe368a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fdc106cd0 con 0x7f6fdc0f5410 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe1426700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe1426700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52036/0 (socket says 172.21.15.97:52036) 2024-02-19T22:20:36.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.250+0000 7f6fe1426700 1 -- 172.21.15.97:0/2111740423 learned_addr learned my addr 172.21.15.97:0/2111740423 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:36.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fe1426700 1 -- 172.21.15.97:0/2111740423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd8005040 con 0x7f6fdc0f5410 2024-02-19T22:20:36.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fe1426700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6fd8005010 tx=0x7f6fd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:36.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8009320 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fdc106ed0 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fd8013380 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fd8014e50 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.251+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fdc107360 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.252+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fd80095f0 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.252+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fdc045ad0 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.252+0000 7f6fd1ffb700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 0x7f6fc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.252+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fd8040e90 con 0x7f6fdc0f5410 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.252+0000 7f6fe0c25700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 0x7f6fc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.253+0000 7f6fe0c25700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 0x7f6fc80323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6fcc0096a0 tx=0x7f6fcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:36.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.258+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fd8014420 con 0x7f6fdc0f5410 2024-02-19T22:20:36.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.461+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fdc0fafa0 con 0x7f6fdc0f5410 2024-02-19T22:20:36.464 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.461+0000 7f6fd1ffb700 1 -- 172.21.15.97:0/2111740423 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6fd8010230 con 0x7f6fdc0f5410 2024-02-19T22:20:36.465 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:36.465 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:36.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 msgr2=0x7f6fc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:36.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 0x7f6fc80323d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f6fcc0096a0 tx=0x7f6fcc006b40).stop 2024-02-19T22:20:36.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 msgr2=0x7f6fdc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6fd8005010 tx=0x7f6fd800b040).stop 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 shutdown_connections 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6fc802ff10 0x7f6fc80323d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 --2- 172.21.15.97:0/2111740423 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6fdc0f5410 0x7f6fdc1067e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 >> 172.21.15.97:0/2111740423 conn(0x7f6fdc0f0b70 msgr2=0x7f6fdc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 shutdown_connections 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:36.463+0000 7f6fe368a700 1 -- 172.21.15.97:0/2111740423 wait complete. 2024-02-19T22:20:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:36.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:36.463458+0000 mon.smithi045 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.97:0/2111740423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:37.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:35.844356+0000 mgr.smithi045.bvlamo (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:38.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:38.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:39.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:37.844699+0000 mgr.smithi045.bvlamo (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- 172.21.15.97:0/242538268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 msgr2=0x7f20280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 --2- 172.21.15.97:0/242538268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20280f6930 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f201c009fb0 tx=0x7f201c009b30).stop 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- 172.21.15.97:0/242538268 shutdown_connections 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 --2- 172.21.15.97:0/242538268 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20280f6930 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- 172.21.15.97:0/242538268 >> 172.21.15.97:0/242538268 conn(0x7f20280f1c70 msgr2=0x7f20280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- 172.21.15.97:0/242538268 shutdown_connections 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- 172.21.15.97:0/242538268 wait complete. 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 Processor -- start 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.339+0000 7f202d00a700 1 -- start start 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f202d00a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:40.345 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f202d00a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028106cd0 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f2026d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f2026d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55364/0 (socket says 172.21.15.97:55364) 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f2026d9d700 1 -- 172.21.15.97:0/3346404040 learned_addr learned my addr 172.21.15.97:0/3346404040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f2026d9d700 1 -- 172.21.15.97:0/3346404040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f201c005040 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.340+0000 7f2026d9d700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f201c0089f0 tx=0x7f201c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201c009320 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f201c013380 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2028106ed0 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f201c01ed80 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2028107360 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2028045ad0 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.341+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f201c01e420 con 0x7f20280f6510 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.342+0000 7f200f7fe700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 0x7f2010032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:40.346 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.342+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f201c040e70 con 0x7f20280f6510 2024-02-19T22:20:40.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.342+0000 7f202659c700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 0x7f2010032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:40.347 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.342+0000 7f202659c700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 0x7f2010032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f20180096a0 tx=0x7f2018006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:40.348 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.346+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f201c00e2a0 con 0x7f20280f6510 2024-02-19T22:20:40.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.555+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2028108210 con 0x7f20280f6510 2024-02-19T22:20:40.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.556+0000 7f200f7fe700 1 -- 172.21.15.97:0/3346404040 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f201c00e2a0 con 0x7f20280f6510 2024-02-19T22:20:40.558 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:40.558 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 msgr2=0x7f2010032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 0x7f2010032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f20180096a0 tx=0x7f2018006b40).stop 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 msgr2=0x7f20281067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f201c0089f0 tx=0x7f201c00b040).stop 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 shutdown_connections 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f201002ff60 0x7f2010032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:40.560 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 --2- 172.21.15.97:0/3346404040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f20280f6510 0x7f20281067e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:40.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 >> 172.21.15.97:0/3346404040 conn(0x7f20280f1c70 msgr2=0x7f20280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:40.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 shutdown_connections 2024-02-19T22:20:40.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:40.558+0000 7f202d00a700 1 -- 172.21.15.97:0/3346404040 wait complete. 2024-02-19T22:20:40.561 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:40.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:40.556966+0000 mon.smithi045 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.97:0/3346404040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:39.845071+0000 mgr.smithi045.bvlamo (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:42.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:42.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:43.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:41.845436+0000 mgr.smithi045.bvlamo (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:44.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 -- 172.21.15.97:0/2325516584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e40f5410 msgr2=0x7f59e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:44.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 --2- 172.21.15.97:0/2325516584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e40f5410 0x7f59e40f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f59d8009fb0 tx=0x7f59d8009b30).stop 2024-02-19T22:20:44.306 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 -- 172.21.15.97:0/2325516584 shutdown_connections 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 --2- 172.21.15.97:0/2325516584 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e40f5410 0x7f59e40f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 -- 172.21.15.97:0/2325516584 >> 172.21.15.97:0/2325516584 conn(0x7f59e40f0b70 msgr2=0x7f59e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 -- 172.21.15.97:0/2325516584 shutdown_connections 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.302+0000 7f59eb064700 1 -- 172.21.15.97:0/2325516584 wait complete. 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59eb064700 1 Processor -- start 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59eb064700 1 -- start start 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59eb064700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59eb064700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4109b30 con 0x7f59e4060200 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59e8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59e8e00700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55374/0 (socket says 172.21.15.97:55374) 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.303+0000 7f59e8e00700 1 -- 172.21.15.97:0/2458307300 learned_addr learned my addr 172.21.15.97:0/2458307300 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59e8e00700 1 -- 172.21.15.97:0/2458307300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8005040 con 0x7f59e4060200 2024-02-19T22:20:44.307 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59e8e00700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f59d8009f80 tx=0x7f59d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:44.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d8013380 con 0x7f59e4060200 2024-02-19T22:20:44.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e4109c70 con 0x7f59e4060200 2024-02-19T22:20:44.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d8013790 con 0x7f59e4060200 2024-02-19T22:20:44.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d8014420 con 0x7f59e4060200 2024-02-19T22:20:44.308 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.304+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e410a0a0 con 0x7f59e4060200 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.305+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59d8025060 con 0x7f59e4060200 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.305+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e4045ad0 con 0x7f59e4060200 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.305+0000 7f59e17fa700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 0x7f59d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.305+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59d8040d70 con 0x7f59e4060200 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.305+0000 7f59e3fff700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 0x7f59d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:44.309 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.306+0000 7f59e3fff700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 0x7f59d4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f59d00096a0 tx=0x7f59d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:44.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.310+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59d801cb60 con 0x7f59e4060200 2024-02-19T22:20:44.516 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.514+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59e410a910 con 0x7f59e4060200 2024-02-19T22:20:44.517 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.515+0000 7f59e17fa700 1 -- 172.21.15.97:0/2458307300 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f59d802d2d0 con 0x7f59e4060200 2024-02-19T22:20:44.517 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:44.517 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:44.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.516+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 msgr2=0x7f59d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.516+0000 7f59eb064700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 0x7f59d4032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f59d00096a0 tx=0x7f59d0006b40).stop 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.516+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 msgr2=0x7f59e4109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.516+0000 7f59eb064700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f59d8009f80 tx=0x7f59d8009b30).stop 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 shutdown_connections 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f59d402ff60 0x7f59d4032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 --2- 172.21.15.97:0/2458307300 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f59e4060200 0x7f59e4109640 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 >> 172.21.15.97:0/2458307300 conn(0x7f59e40f0b70 msgr2=0x7f59e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 shutdown_connections 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:44.517+0000 7f59eb064700 1 -- 172.21.15.97:0/2458307300 wait complete. 2024-02-19T22:20:44.519 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:44.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:44.516011+0000 mon.smithi045 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.97:0/2458307300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:45.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:43.845778+0000 mgr.smithi045.bvlamo (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:46.189 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:46.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:45.846134+0000 mgr.smithi045.bvlamo (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:48.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- 172.21.15.97:0/2586171186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 msgr2=0x7f30cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:48.418 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 --2- 172.21.15.97:0/2586171186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc0f66c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f30bc009fb0 tx=0x7f30bc009b30).stop 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- 172.21.15.97:0/2586171186 shutdown_connections 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 --2- 172.21.15.97:0/2586171186 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc0f66c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- 172.21.15.97:0/2586171186 >> 172.21.15.97:0/2586171186 conn(0x7f30cc0f1ac0 msgr2=0x7f30cc0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- 172.21.15.97:0/2586171186 shutdown_connections 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- 172.21.15.97:0/2586171186 wait complete. 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 Processor -- start 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- start start 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.415+0000 7f30d255e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30cc106ab0 con 0x7f30cc0f62a0 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d155c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d155c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35590/0 (socket says 172.21.15.97:35590) 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d155c700 1 -- 172.21.15.97:0/3587514671 learned_addr learned my addr 172.21.15.97:0/3587514671 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:48.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d155c700 1 -- 172.21.15.97:0/3587514671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30bc005040 con 0x7f30cc0f62a0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d155c700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f30bc0089f0 tx=0x7f30bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30bc009320 con 0x7f30cc0f62a0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30cc106cb0 con 0x7f30cc0f62a0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f30bc013380 con 0x7f30cc0f62a0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30bc014e20 con 0x7f30cc0f62a0 2024-02-19T22:20:48.420 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.416+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30cc107140 con 0x7f30cc0f62a0 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.417+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30bc0095f0 con 0x7f30cc0f62a0 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.417+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30cc045ad0 con 0x7f30cc0f62a0 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.417+0000 7f30c1ffb700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 0x7f30b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.417+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30bc040f50 con 0x7f30cc0f62a0 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.418+0000 7f30d0d5b700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 0x7f30b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:48.421 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.418+0000 7f30d0d5b700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 0x7f30b80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f30c80096a0 tx=0x7f30c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:48.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.422+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30bc014420 con 0x7f30cc0f62a0 2024-02-19T22:20:48.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.626+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30cc107ff0 con 0x7f30cc0f62a0 2024-02-19T22:20:48.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.627+0000 7f30c1ffb700 1 -- 172.21.15.97:0/3587514671 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f30bc01d980 con 0x7f30cc0f62a0 2024-02-19T22:20:48.629 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:48.629 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.628+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 msgr2=0x7f30b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.628+0000 7f30d255e700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 0x7f30b80323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f30c80096a0 tx=0x7f30c8006b40).stop 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.628+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 msgr2=0x7f30cc1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.628+0000 7f30d255e700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f30bc0089f0 tx=0x7f30bc00b040).stop 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 shutdown_connections 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f30b802ff10 0x7f30b80323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 --2- 172.21.15.97:0/3587514671 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f30cc0f62a0 0x7f30cc1065c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 >> 172.21.15.97:0/3587514671 conn(0x7f30cc0f1ac0 msgr2=0x7f30cc0f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:48.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 shutdown_connections 2024-02-19T22:20:48.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:48.629+0000 7f30d255e700 1 -- 172.21.15.97:0/3587514671 wait complete. 2024-02-19T22:20:48.632 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:47.846419+0000 mgr.smithi045.bvlamo (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:48.627871+0000 mon.smithi045 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.97:0/3587514671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:50.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:50.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:49.846738+0000 mgr.smithi045.bvlamo (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:52.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- 172.21.15.97:0/1068048850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 msgr2=0x7f341c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:52.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 --2- 172.21.15.97:0/1068048850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f3418009fb0 tx=0x7f3418009b30).stop 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- 172.21.15.97:0/1068048850 shutdown_connections 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 --2- 172.21.15.97:0/1068048850 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- 172.21.15.97:0/1068048850 >> 172.21.15.97:0/1068048850 conn(0x7f341c0f0b70 msgr2=0x7f341c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- 172.21.15.97:0/1068048850 shutdown_connections 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- 172.21.15.97:0/1068048850 wait complete. 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 Processor -- start 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- start start 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.296+0000 7f342379b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341c105bd0 con 0x7f341c0f5410 2024-02-19T22:20:52.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3421537700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3421537700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35610/0 (socket says 172.21.15.97:35610) 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3421537700 1 -- 172.21.15.97:0/1912662296 learned_addr learned my addr 172.21.15.97:0/1912662296 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3421537700 1 -- 172.21.15.97:0/1912662296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3418005040 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3421537700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f34180089f0 tx=0x7f341800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3418009320 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f341c105dd0 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3418013380 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f341801ed80 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.297+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f341c106260 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.298+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f341801e420 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.298+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f341c045ad0 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.298+0000 7f3411ffb700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 0x7f3408032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.299+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3418041030 con 0x7f341c0f5410 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.299+0000 7f3420d36700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 0x7f3408032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:52.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.299+0000 7f3420d36700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 0x7f3408032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f340c0096a0 tx=0x7f340c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:52.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.304+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f341801cb10 con 0x7f341c0f5410 2024-02-19T22:20:52.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.509+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f341c1070b0 con 0x7f341c0f5410 2024-02-19T22:20:52.511 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.509+0000 7f3411ffb700 1 -- 172.21.15.97:0/1912662296 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f341802c2d0 con 0x7f341c0f5410 2024-02-19T22:20:52.511 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:52.512 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:52.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 msgr2=0x7f3408032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:52.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 0x7f3408032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f340c0096a0 tx=0x7f340c006b40).stop 2024-02-19T22:20:52.513 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 msgr2=0x7f341c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f34180089f0 tx=0x7f341800b040).stop 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 shutdown_connections 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f340802ff60 0x7f3408032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 --2- 172.21.15.97:0/1912662296 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f341c0f5410 0x7f341c1056e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 >> 172.21.15.97:0/1912662296 conn(0x7f341c0f0b70 msgr2=0x7f341c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 shutdown_connections 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:52.511+0000 7f342379b700 1 -- 172.21.15.97:0/1912662296 wait complete. 2024-02-19T22:20:52.514 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:52.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:52.510569+0000 mon.smithi045 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.97:0/1912662296' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:53.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:51.847088+0000 mgr.smithi045.bvlamo (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:54.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:54.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:53.847389+0000 mgr.smithi045.bvlamo (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:56.422 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 -- 172.21.15.97:0/3608434719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 msgr2=0x7fcd0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 --2- 172.21.15.97:0/3608434719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f90f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fccfc009fb0 tx=0x7fccfc009b30).stop 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 -- 172.21.15.97:0/3608434719 shutdown_connections 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 --2- 172.21.15.97:0/3608434719 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f90f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 -- 172.21.15.97:0/3608434719 >> 172.21.15.97:0/3608434719 conn(0x7fcd0c0f0b70 msgr2=0x7fcd0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 -- 172.21.15.97:0/3608434719 shutdown_connections 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 -- 172.21.15.97:0/3608434719 wait complete. 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.412+0000 7fcd1280c700 1 Processor -- start 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd1280c700 1 -- start start 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd1280c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd1280c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0f4ff0 con 0x7fcd0c060280 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd0bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:56.423 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd0bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54972/0 (socket says 172.21.15.97:54972) 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd0bfff700 1 -- 172.21.15.97:0/387624950 learned_addr learned my addr 172.21.15.97:0/387624950 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd0bfff700 1 -- 172.21.15.97:0/387624950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc005040 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.413+0000 7fcd0bfff700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fccfc0089f0 tx=0x7fccfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccfc009320 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccfc013380 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c0f5190 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccfc014e20 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c0f5620 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.414+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccfc0095f0 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.415+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c059100 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.415+0000 7fcd08ff9700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 0x7fccf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.415+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccfc040e40 con 0x7fcd0c060280 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.415+0000 7fcd0b7fe700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 0x7fccf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:56.424 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.415+0000 7fcd0b7fe700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 0x7fccf4032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcd000096a0 tx=0x7fcd00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:56.425 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.420+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccfc014420 con 0x7fcd0c060280 2024-02-19T22:20:56.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.627+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd0c0fc090 con 0x7fcd0c060280 2024-02-19T22:20:56.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.628+0000 7fcd08ff9700 1 -- 172.21.15.97:0/387624950 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccfc010230 con 0x7fcd0c060280 2024-02-19T22:20:56.630 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:20:56.630 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.629+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 msgr2=0x7fccf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.629+0000 7fcd1280c700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 0x7fccf4032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fcd000096a0 tx=0x7fcd00006b40).stop 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 msgr2=0x7fcd0c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fccfc0089f0 tx=0x7fccfc00b040).stop 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 shutdown_connections 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccf402ff60 0x7fccf4032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 --2- 172.21.15.97:0/387624950 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd0c060280 0x7fcd0c0f6b70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 >> 172.21.15.97:0/387624950 conn(0x7fcd0c0f0b70 msgr2=0x7fcd0c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:56.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 shutdown_connections 2024-02-19T22:20:56.633 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:20:56.630+0000 7fcd1280c700 1 -- 172.21.15.97:0/387624950 wait complete. 2024-02-19T22:20:56.633 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:20:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:55.847683+0000 mgr.smithi045.bvlamo (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:20:56.629077+0000 mon.smithi045 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.97:0/387624950' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:58.357 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:58.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:20:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:57.847980+0000 mgr.smithi045.bvlamo (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 -- 172.21.15.97:0/1030766897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 msgr2=0x7f71d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 --2- 172.21.15.97:0/1030766897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d00f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f71c0009fb0 tx=0x7f71c0009b30).stop 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 -- 172.21.15.97:0/1030766897 shutdown_connections 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 --2- 172.21.15.97:0/1030766897 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d00f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 -- 172.21.15.97:0/1030766897 >> 172.21.15.97:0/1030766897 conn(0x7f71d00f1c70 msgr2=0x7f71d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.310+0000 7f71d58b2700 1 -- 172.21.15.97:0/1030766897 shutdown_connections 2024-02-19T22:21:00.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.311+0000 7f71d58b2700 1 -- 172.21.15.97:0/1030766897 wait complete. 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.311+0000 7f71d58b2700 1 Processor -- start 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.311+0000 7f71d58b2700 1 -- start start 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71d58b2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71d58b2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0106cd0 con 0x7f71d00f6510 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71ceffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71ceffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55000/0 (socket says 172.21.15.97:55000) 2024-02-19T22:21:00.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71ceffd700 1 -- 172.21.15.97:0/3730576639 learned_addr learned my addr 172.21.15.97:0/3730576639 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71ceffd700 1 -- 172.21.15.97:0/3730576639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c0005040 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71ceffd700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f71c0005010 tx=0x7f71c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c0009320 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d0106ed0 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71c0013380 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c0014e50 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.312+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0107360 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.313+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71c00095f0 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.313+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0045ad0 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.313+0000 7f71b77fe700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 0x7f71b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.313+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71c0040fa0 con 0x7f71d00f6510 2024-02-19T22:21:00.317 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.314+0000 7f71ce7fc700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 0x7f71b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:00.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.314+0000 7f71ce7fc700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 0x7f71b80323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f71c40096a0 tx=0x7f71c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:00.320 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.318+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71c0014420 con 0x7f71d00f6510 2024-02-19T22:21:00.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.525+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71d0108210 con 0x7f71d00f6510 2024-02-19T22:21:00.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.526+0000 7f71b77fe700 1 -- 172.21.15.97:0/3730576639 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71c0010230 con 0x7f71d00f6510 2024-02-19T22:21:00.529 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:00.530 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:00.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.527+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 msgr2=0x7f71b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 0x7f71b80323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f71c40096a0 tx=0x7f71c4006b40).stop 2024-02-19T22:21:00.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 msgr2=0x7f71d01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:00.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f71c0005010 tx=0x7f71c000b040).stop 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 shutdown_connections 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f71b802ff10 0x7f71b80323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 --2- 172.21.15.97:0/3730576639 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71d00f6510 0x7f71d01067e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 >> 172.21.15.97:0/3730576639 conn(0x7f71d00f1c70 msgr2=0x7f71d00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 shutdown_connections 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:00.528+0000 7f71d58b2700 1 -- 172.21.15.97:0/3730576639 wait complete. 2024-02-19T22:21:00.531 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:00.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:00.527123+0000 mon.smithi045 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.97:0/3730576639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:20:59.848276+0000 mgr.smithi045.bvlamo (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:02.350 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:02.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:01.848671+0000 mgr.smithi045.bvlamo (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- 172.21.15.97:0/3647726346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 msgr2=0x7f48340f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 --2- 172.21.15.97:0/3647726346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f4830009fb0 tx=0x7f4830009b30).stop 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- 172.21.15.97:0/3647726346 shutdown_connections 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 --2- 172.21.15.97:0/3647726346 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- 172.21.15.97:0/3647726346 >> 172.21.15.97:0/3647726346 conn(0x7f48340f09a0 msgr2=0x7f48340f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- 172.21.15.97:0/3647726346 shutdown_connections 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- 172.21.15.97:0/3647726346 wait complete. 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 Processor -- start 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.259+0000 7f483a974700 1 -- start start 2024-02-19T22:21:04.263 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f483a974700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f483a974700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48340fb6f0 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f4839972700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f4839972700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55026/0 (socket says 172.21.15.97:55026) 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f4839972700 1 -- 172.21.15.97:0/3467244262 learned_addr learned my addr 172.21.15.97:0/3467244262 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f4839972700 1 -- 172.21.15.97:0/3467244262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830005040 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f4839972700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f48300089f0 tx=0x7f483000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4830009320 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48340fb890 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4830013380 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4830014e20 con 0x7f48340f51a0 2024-02-19T22:21:04.264 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.260+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48340fbd20 con 0x7f48340f51a0 2024-02-19T22:21:04.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.261+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48300095f0 con 0x7f48340f51a0 2024-02-19T22:21:04.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.261+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4834059100 con 0x7f48340f51a0 2024-02-19T22:21:04.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.261+0000 7f482a7fc700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:04.265 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.261+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4830040ec0 con 0x7f48340f51a0 2024-02-19T22:21:04.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.262+0000 7f4839171700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:04.266 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.262+0000 7f4839171700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 0x7f4820032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:04.268 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.266+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4830014420 con 0x7f48340f51a0 2024-02-19T22:21:04.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.472+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48340fad30 con 0x7f48340f51a0 2024-02-19T22:21:04.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.473+0000 7f482a7fc700 1 -- 172.21.15.97:0/3467244262 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f483001d980 con 0x7f48340f51a0 2024-02-19T22:21:04.475 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:04.475 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 msgr2=0x7f4820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 0x7f4820032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f48240096a0 tx=0x7f4824006b40).stop 2024-02-19T22:21:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 msgr2=0x7f48340fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f48300089f0 tx=0x7f483000b040).stop 2024-02-19T22:21:04.476 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 shutdown_connections 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 --2- 172.21.15.97:0/3467244262 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f48340f51a0 0x7f48340fd270 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 >> 172.21.15.97:0/3467244262 conn(0x7f48340f09a0 msgr2=0x7f483405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 shutdown_connections 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:04.474+0000 7f483a974700 1 -- 172.21.15.97:0/3467244262 wait complete. 2024-02-19T22:21:04.477 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:03.936341+0000 mon.smithi045 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:21:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:03.937062+0000 mon.smithi045 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:21:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:04.473707+0000 mon.smithi045 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.97:0/3467244262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:03.849060+0000 mgr.smithi045.bvlamo (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:06.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:06.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:07.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:05.849329+0000 mgr.smithi045.bvlamo (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:08.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 -- 172.21.15.97:0/160325611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 msgr2=0x7f90340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 --2- 172.21.15.97:0/160325611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90340f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f9024009fb0 tx=0x7f9024009b30).stop 2024-02-19T22:21:08.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 -- 172.21.15.97:0/160325611 shutdown_connections 2024-02-19T22:21:08.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 --2- 172.21.15.97:0/160325611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90340f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.310 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 -- 172.21.15.97:0/160325611 >> 172.21.15.97:0/160325611 conn(0x7f90340f0b70 msgr2=0x7f90340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 -- 172.21.15.97:0/160325611 shutdown_connections 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.306+0000 7f9039df2700 1 -- 172.21.15.97:0/160325611 wait complete. 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f9039df2700 1 Processor -- start 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f9039df2700 1 -- start start 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f9039df2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f9039df2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9034105bd0 con 0x7f90340f5410 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f90337fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f90337fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35398/0 (socket says 172.21.15.97:35398) 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.307+0000 7f90337fe700 1 -- 172.21.15.97:0/2529185881 learned_addr learned my addr 172.21.15.97:0/2529185881 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f90337fe700 1 -- 172.21.15.97:0/2529185881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9024005040 con 0x7f90340f5410 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f90337fe700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f9024005010 tx=0x7f902400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9024009320 con 0x7f90340f5410 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9034105dd0 con 0x7f90340f5410 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9024013380 con 0x7f90340f5410 2024-02-19T22:21:08.311 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9024014e50 con 0x7f90340f5410 2024-02-19T22:21:08.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.308+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9034106260 con 0x7f90340f5410 2024-02-19T22:21:08.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.309+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90240263a0 con 0x7f90340f5410 2024-02-19T22:21:08.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.309+0000 7f902bfff700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 0x7f90200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:08.312 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.309+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9024040fa0 con 0x7f90340f5410 2024-02-19T22:21:08.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.309+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9034045ad0 con 0x7f90340f5410 2024-02-19T22:21:08.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.309+0000 7f9032ffd700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 0x7f90200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:08.313 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.310+0000 7f9032ffd700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 0x7f90200323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f901c0096a0 tx=0x7f901c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:08.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.314+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9024012920 con 0x7f90340f5410 2024-02-19T22:21:08.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.518+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9034106fb0 con 0x7f90340f5410 2024-02-19T22:21:08.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.518+0000 7f902bfff700 1 -- 172.21.15.97:0/2529185881 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f902402b020 con 0x7f90340f5410 2024-02-19T22:21:08.521 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:08.521 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:08.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 msgr2=0x7f90200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 0x7f90200323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f901c0096a0 tx=0x7f901c006b40).stop 2024-02-19T22:21:08.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 msgr2=0x7f90341056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:08.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f9024005010 tx=0x7f902400b040).stop 2024-02-19T22:21:08.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 shutdown_connections 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f902002ff10 0x7f90200323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 --2- 172.21.15.97:0/2529185881 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f90340f5410 0x7f90341056e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 >> 172.21.15.97:0/2529185881 conn(0x7f90340f0b70 msgr2=0x7f90340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 shutdown_connections 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:08.520+0000 7f9039df2700 1 -- 172.21.15.97:0/2529185881 wait complete. 2024-02-19T22:21:08.523 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:08.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:08.519364+0000 mon.smithi045 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.97:0/2529185881' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:09.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:07.849659+0000 mgr.smithi045.bvlamo (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:10.112 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:10.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:09.849939+0000 mgr.smithi045.bvlamo (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:12.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.186+0000 7f695030f700 1 -- 172.21.15.97:0/2167104306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 msgr2=0x7f69480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.186+0000 7f695030f700 1 --2- 172.21.15.97:0/2167104306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f69480f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f6940009960 tx=0x7f6940004d10).stop 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.186+0000 7f695030f700 1 -- 172.21.15.97:0/2167104306 shutdown_connections 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 --2- 172.21.15.97:0/2167104306 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f69480f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 -- 172.21.15.97:0/2167104306 >> 172.21.15.97:0/2167104306 conn(0x7f69480f1c70 msgr2=0x7f69480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 -- 172.21.15.97:0/2167104306 shutdown_connections 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 -- 172.21.15.97:0/2167104306 wait complete. 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 Processor -- start 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 -- start start 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f695030f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948106d10 con 0x7f69480f6510 2024-02-19T22:21:12.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f694e0ab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f694e0ab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35410/0 (socket says 172.21.15.97:35410) 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.187+0000 7f694e0ab700 1 -- 172.21.15.97:0/1834301588 learned_addr learned my addr 172.21.15.97:0/1834301588 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f694e0ab700 1 -- 172.21.15.97:0/1834301588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6940009640 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f694e0ab700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6940004a10 tx=0x7f6940004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6940018070 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6940007e70 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6948106f10 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f694001d420 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.188+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69481073a0 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.189+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f694001d6e0 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.189+0000 7f6946ffd700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.189+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6940041610 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.189+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6948045ad0 con 0x7f69480f6510 2024-02-19T22:21:12.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.189+0000 7f694d8aa700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:12.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.190+0000 7f694d8aa700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 0x7f6930032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:12.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.194+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f694001dae0 con 0x7f69480f6510 2024-02-19T22:21:12.400 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.399+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69480fc190 con 0x7f69480f6510 2024-02-19T22:21:12.401 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.399+0000 7f6946ffd700 1 -- 172.21.15.97:0/1834301588 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6940011320 con 0x7f69480f6510 2024-02-19T22:21:12.401 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:12.402 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:12.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 msgr2=0x7f6930032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 0x7f6930032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f693c0096a0 tx=0x7f693c006b40).stop 2024-02-19T22:21:12.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 msgr2=0x7f6948106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:12.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6940004a10 tx=0x7f6940004d10).stop 2024-02-19T22:21:12.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 shutdown_connections 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f693002ff60 0x7f6930032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 --2- 172.21.15.97:0/1834301588 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f69480f6510 0x7f6948106820 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 >> 172.21.15.97:0/1834301588 conn(0x7f69480f1c70 msgr2=0x7f69480f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 shutdown_connections 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:12.401+0000 7f695030f700 1 -- 172.21.15.97:0/1834301588 wait complete. 2024-02-19T22:21:12.404 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:12.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:12 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:12.400435+0000 mon.smithi045 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.97:0/1834301588' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:11.850300+0000 mgr.smithi045.bvlamo (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:13.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:13.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:13.850694+0000 mgr.smithi045.bvlamo (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 -- 172.21.15.97:0/347120122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 msgr2=0x7f03780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 --2- 172.21.15.97:0/347120122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f03780f5830 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f0360009fb0 tx=0x7f0360009b30).stop 2024-02-19T22:21:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 -- 172.21.15.97:0/347120122 shutdown_connections 2024-02-19T22:21:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 --2- 172.21.15.97:0/347120122 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f03780f5830 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 -- 172.21.15.97:0/347120122 >> 172.21.15.97:0/347120122 conn(0x7f03780f0b70 msgr2=0x7f03780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 -- 172.21.15.97:0/347120122 shutdown_connections 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.111+0000 7f037db86700 1 -- 172.21.15.97:0/347120122 wait complete. 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f037db86700 1 Processor -- start 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f037db86700 1 -- start start 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f037db86700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f037db86700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0378107330 con 0x7f03780f5410 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f03777fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f03777fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35430/0 (socket says 172.21.15.97:35430) 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.112+0000 7f03777fe700 1 -- 172.21.15.97:0/4012753421 learned_addr learned my addr 172.21.15.97:0/4012753421 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f03777fe700 1 -- 172.21.15.97:0/4012753421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360005040 con 0x7f03780f5410 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f03777fe700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0360005010 tx=0x7f036000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360009320 con 0x7f03780f5410 2024-02-19T22:21:16.116 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0378060220 con 0x7f03780f5410 2024-02-19T22:21:16.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0360013380 con 0x7f03780f5410 2024-02-19T22:21:16.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0360014e50 con 0x7f03780f5410 2024-02-19T22:21:16.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.113+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037810be30 con 0x7f03780f5410 2024-02-19T22:21:16.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.114+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03600095f0 con 0x7f03780f5410 2024-02-19T22:21:16.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.114+0000 7f036ffff700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 0x7f03640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:16.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.114+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0378045ad0 con 0x7f03780f5410 2024-02-19T22:21:16.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.114+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0360040f10 con 0x7f03780f5410 2024-02-19T22:21:16.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.114+0000 7f0376ffd700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 0x7f03640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:16.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.115+0000 7f0376ffd700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 0x7f03640323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f03680096a0 tx=0x7f0368006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:16.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.119+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0360014420 con 0x7f03780f5410 2024-02-19T22:21:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.331+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f037810c620 con 0x7f03780f5410 2024-02-19T22:21:16.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.331+0000 7f036ffff700 1 -- 172.21.15.97:0/4012753421 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f036002c090 con 0x7f03780f5410 2024-02-19T22:21:16.334 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:16.334 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 msgr2=0x7f03640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 0x7f03640323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f03680096a0 tx=0x7f0368006b40).stop 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 msgr2=0x7f037810b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f0360005010 tx=0x7f036000b040).stop 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 shutdown_connections 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f036402ff10 0x7f03640323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 --2- 172.21.15.97:0/4012753421 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f03780f5410 0x7f037810b940 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 >> 172.21.15.97:0/4012753421 conn(0x7f03780f0b70 msgr2=0x7f03780f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 shutdown_connections 2024-02-19T22:21:16.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:16.333+0000 7f037db86700 1 -- 172.21.15.97:0/4012753421 wait complete. 2024-02-19T22:21:16.336 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:16.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:16.332403+0000 mon.smithi045 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.97:0/4012753421' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:15.851075+0000 mgr.smithi045.bvlamo (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:17.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:17.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:19.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:17.851468+0000 mgr.smithi045.bvlamo (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 -- 172.21.15.97:0/1435929866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c0f6510 msgr2=0x7f0e6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1435929866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c0f6510 0x7f0e6c0f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f0e60009fb0 tx=0x7f0e60009b30).stop 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 -- 172.21.15.97:0/1435929866 shutdown_connections 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1435929866 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c0f6510 0x7f0e6c0f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 -- 172.21.15.97:0/1435929866 >> 172.21.15.97:0/1435929866 conn(0x7f0e6c0f1c70 msgr2=0x7f0e6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 -- 172.21.15.97:0/1435929866 shutdown_connections 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.034+0000 7f0e735b8700 1 -- 172.21.15.97:0/1435929866 wait complete. 2024-02-19T22:21:20.086 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e735b8700 1 Processor -- start 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e735b8700 1 -- start start 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e735b8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e735b8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c107210 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e71354700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e71354700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52668/0 (socket says 172.21.15.97:52668) 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.035+0000 7f0e71354700 1 -- 172.21.15.97:0/1678440844 learned_addr learned my addr 172.21.15.97:0/1678440844 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e71354700 1 -- 172.21.15.97:0/1678440844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e60005040 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e71354700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0e60009f80 tx=0x7f0e60009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e60013380 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c107410 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e60013790 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e60014420 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.036+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c107f60 con 0x7f0e6c106900 2024-02-19T22:21:20.087 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.037+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e60025060 con 0x7f0e6c106900 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.037+0000 7f0e5dffb700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.037+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c059100 con 0x7f0e6c106900 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.037+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e600411b0 con 0x7f0e6c106900 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.037+0000 7f0e70b53700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.038+0000 7f0e70b53700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 0x7f0e58032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0e680096a0 tx=0x7f0e68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:20.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.042+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e60028440 con 0x7f0e6c106900 2024-02-19T22:21:20.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.254+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e6c045ad0 con 0x7f0e6c106900 2024-02-19T22:21:20.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.254+0000 7f0e5dffb700 1 -- 172.21.15.97:0/1678440844 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e6003d090 con 0x7f0e6c106900 2024-02-19T22:21:20.257 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:20.257 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 msgr2=0x7f0e58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 0x7f0e58032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0e680096a0 tx=0x7f0e68006b40).stop 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 msgr2=0x7f0e6c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f0e60009f80 tx=0x7f0e60009b30).stop 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 shutdown_connections 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e5802ff60 0x7f0e58032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 --2- 172.21.15.97:0/1678440844 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e6c106900 0x7f0e6c106d20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:20.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 >> 172.21.15.97:0/1678440844 conn(0x7f0e6c0f1c70 msgr2=0x7f0e6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 shutdown_connections 2024-02-19T22:21:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:20.256+0000 7f0e735b8700 1 -- 172.21.15.97:0/1678440844 wait complete. 2024-02-19T22:21:20.259 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:20.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:20.255412+0000 mon.smithi045 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.97:0/1678440844' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:21.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:19.851716+0000 mgr.smithi045.bvlamo (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:21.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:21.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:23.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 -- 172.21.15.97:0/4033389137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 msgr2=0x7f9df40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:23.874 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 --2- 172.21.15.97:0/4033389137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df40f66c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f9de8009fb0 tx=0x7f9de8009b30).stop 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 -- 172.21.15.97:0/4033389137 shutdown_connections 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 --2- 172.21.15.97:0/4033389137 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df40f66c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 -- 172.21.15.97:0/4033389137 >> 172.21.15.97:0/4033389137 conn(0x7f9df40f1aa0 msgr2=0x7f9df40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 -- 172.21.15.97:0/4033389137 shutdown_connections 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 -- 172.21.15.97:0/4033389137 wait complete. 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.872+0000 7f9df904c700 1 Processor -- start 2024-02-19T22:21:23.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df904c700 1 -- start start 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df904c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df904c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df4069e20 con 0x7f9df40f62a0 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52686/0 (socket says 172.21.15.97:52686) 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df3fff700 1 -- 172.21.15.97:0/2387131570 learned_addr learned my addr 172.21.15.97:0/2387131570 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df3fff700 1 -- 172.21.15.97:0/2387131570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de8005040 con 0x7f9df40f62a0 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.873+0000 7f9df3fff700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9de80089f0 tx=0x7f9de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:23.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de8009320 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df4069fc0 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9de8013380 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9de8014e20 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df406a450 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9de8014420 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.874+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df4059100 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.875+0000 7f9df0ff9700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 0x7f9ddc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.875+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9de8040f50 con 0x7f9df40f62a0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.875+0000 7f9df37fe700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 0x7f9ddc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:23.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.875+0000 7f9df37fe700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 0x7f9ddc0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9de40096a0 tx=0x7f9de4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:23.881 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:23.879+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9de806d2b0 con 0x7f9df40f62a0 2024-02-19T22:21:23.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:21.852061+0000 mgr.smithi045.bvlamo (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:24.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.086+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9df40fbe30 con 0x7f9df40f62a0 2024-02-19T22:21:24.089 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.087+0000 7f9df0ff9700 1 -- 172.21.15.97:0/2387131570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9de80095f0 con 0x7f9df40f62a0 2024-02-19T22:21:24.089 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:24.089 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 msgr2=0x7f9ddc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 0x7f9ddc0323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f9de40096a0 tx=0x7f9de4006b40).stop 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 msgr2=0x7f9df406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f9de80089f0 tx=0x7f9de800b040).stop 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 shutdown_connections 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ddc02ff10 0x7f9ddc0323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 --2- 172.21.15.97:0/2387131570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9df40f62a0 0x7f9df406b9a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 >> 172.21.15.97:0/2387131570 conn(0x7f9df40f1aa0 msgr2=0x7f9df40f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 shutdown_connections 2024-02-19T22:21:24.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:24.088+0000 7f9df904c700 1 -- 172.21.15.97:0/2387131570 wait complete. 2024-02-19T22:21:24.092 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:24.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:24.087865+0000 mon.smithi045 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.97:0/2387131570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:25.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:25.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:25.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:23.852352+0000 mgr.smithi045.bvlamo (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:25.852626+0000 mgr.smithi045.bvlamo (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:27.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- 172.21.15.97:0/2916985453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 msgr2=0x7ff7600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:27.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 --2- 172.21.15.97:0/2916985453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600f5830 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7ff754009fb0 tx=0x7ff754009b30).stop 2024-02-19T22:21:27.931 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- 172.21.15.97:0/2916985453 shutdown_connections 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 --2- 172.21.15.97:0/2916985453 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600f5830 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- 172.21.15.97:0/2916985453 >> 172.21.15.97:0/2916985453 conn(0x7ff7600f0b70 msgr2=0x7ff7600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- 172.21.15.97:0/2916985453 shutdown_connections 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- 172.21.15.97:0/2916985453 wait complete. 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 Processor -- start 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- start start 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.928+0000 7ff767e52700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7600fb960 con 0x7ff7600f5410 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff765bee700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff765bee700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37576/0 (socket says 172.21.15.97:37576) 2024-02-19T22:21:27.932 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff765bee700 1 -- 172.21.15.97:0/3689828142 learned_addr learned my addr 172.21.15.97:0/3689828142 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff765bee700 1 -- 172.21.15.97:0/3689828142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754005040 con 0x7ff7600f5410 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff765bee700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7ff754005010 tx=0x7ff75400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff754009320 con 0x7ff7600f5410 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7600fbb00 con 0x7ff7600f5410 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff754013380 con 0x7ff7600f5410 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff754014e50 con 0x7ff7600f5410 2024-02-19T22:21:27.933 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.929+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7600fbf90 con 0x7ff7600f5410 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.930+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7540095f0 con 0x7ff7600f5410 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.930+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff760059100 con 0x7ff7600f5410 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.930+0000 7ff7527fc700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 0x7ff74c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.930+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff754040fa0 con 0x7ff7600f5410 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.930+0000 7ff7653ed700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 0x7ff74c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:27.934 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.931+0000 7ff7653ed700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 0x7ff74c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff75c0096a0 tx=0x7ff75c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:27.937 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:27.935+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff754014420 con 0x7ff7600f5410 2024-02-19T22:21:28.141 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.139+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7600fafa0 con 0x7ff7600f5410 2024-02-19T22:21:28.142 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.140+0000 7ff7527fc700 1 -- 172.21.15.97:0/3689828142 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff754010230 con 0x7ff7600f5410 2024-02-19T22:21:28.142 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:28.143 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 msgr2=0x7ff74c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 0x7ff74c0323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7ff75c0096a0 tx=0x7ff75c006b40).stop 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 msgr2=0x7ff7600fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7ff754005010 tx=0x7ff75400b040).stop 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 shutdown_connections 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff74c02ff10 0x7ff74c0323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 --2- 172.21.15.97:0/3689828142 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7600f5410 0x7ff7600fd4e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 >> 172.21.15.97:0/3689828142 conn(0x7ff7600f0b70 msgr2=0x7ff76005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 shutdown_connections 2024-02-19T22:21:28.144 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:28.142+0000 7ff767e52700 1 -- 172.21.15.97:0/3689828142 wait complete. 2024-02-19T22:21:28.145 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:28.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:28.141166+0000 mon.smithi045 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.97:0/3689828142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:29.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:29.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:29.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:27.852998+0000 mgr.smithi045.bvlamo (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:29.853376+0000 mgr.smithi045.bvlamo (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 -- 172.21.15.97:0/243865182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 msgr2=0x7fc4a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 --2- 172.21.15.97:0/243865182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a00f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fc494009fb0 tx=0x7fc494009b30).stop 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 -- 172.21.15.97:0/243865182 shutdown_connections 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 --2- 172.21.15.97:0/243865182 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a00f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 -- 172.21.15.97:0/243865182 >> 172.21.15.97:0/243865182 conn(0x7fc4a00f1c70 msgr2=0x7fc4a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:31.750 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 -- 172.21.15.97:0/243865182 shutdown_connections 2024-02-19T22:21:31.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.747+0000 7fc4a573b700 1 -- 172.21.15.97:0/243865182 wait complete. 2024-02-19T22:21:31.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc4a573b700 1 Processor -- start 2024-02-19T22:21:31.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc4a573b700 1 -- start start 2024-02-19T22:21:31.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc4a573b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:31.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc4a573b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0108ef0 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc49effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc49effd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37592/0 (socket says 172.21.15.97:37592) 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.748+0000 7fc49effd700 1 -- 172.21.15.97:0/2683541491 learned_addr learned my addr 172.21.15.97:0/2683541491 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc49effd700 1 -- 172.21.15.97:0/2683541491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc494005040 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc49effd700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc494005010 tx=0x7fc49400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc494009320 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a01090f0 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc494013380 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc494014e50 con 0x7fc4a00f6510 2024-02-19T22:21:31.753 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.749+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a0109580 con 0x7fc4a00f6510 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc494014420 con 0x7fc4a00f6510 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0045ad0 con 0x7fc4a00f6510 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc4877fe700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 0x7fc4880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc494040f40 con 0x7fc4a00f6510 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc49e7fc700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 0x7fc4880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:31.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.750+0000 7fc49e7fc700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 0x7fc4880323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc4900096a0 tx=0x7fc490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:31.757 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.755+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4940095f0 con 0x7fc4a00f6510 2024-02-19T22:21:31.960 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.958+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4a00fc0a0 con 0x7fc4a00f6510 2024-02-19T22:21:31.961 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.959+0000 7fc4877fe700 1 -- 172.21.15.97:0/2683541491 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc49401db70 con 0x7fc4a00f6510 2024-02-19T22:21:31.961 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:31.961 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:31.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 msgr2=0x7fc4880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 0x7fc4880323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fc4900096a0 tx=0x7fc490006b40).stop 2024-02-19T22:21:31.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 msgr2=0x7fc4a0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:31.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fc494005010 tx=0x7fc49400b040).stop 2024-02-19T22:21:31.962 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 shutdown_connections 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc48802ff10 0x7fc4880323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 --2- 172.21.15.97:0/2683541491 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc4a00f6510 0x7fc4a0108a00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 >> 172.21.15.97:0/2683541491 conn(0x7fc4a00f1c70 msgr2=0x7fc4a00f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 shutdown_connections 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:31.960+0000 7fc4a573b700 1 -- 172.21.15.97:0/2683541491 wait complete. 2024-02-19T22:21:31.963 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:33.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:31.853786+0000 mgr.smithi045.bvlamo (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:33.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:31.959733+0000 mon.smithi045 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.97:0/2683541491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:33.743 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:33.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:35.182 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:33.854213+0000 mgr.smithi045.bvlamo (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 -- 172.21.15.97:0/453538834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 msgr2=0x7ff7180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 --2- 172.21.15.97:0/453538834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7180f5830 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7ff708009fb0 tx=0x7ff708009b30).stop 2024-02-19T22:21:35.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 -- 172.21.15.97:0/453538834 shutdown_connections 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 --2- 172.21.15.97:0/453538834 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7180f5830 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 -- 172.21.15.97:0/453538834 >> 172.21.15.97:0/453538834 conn(0x7ff7180f0b70 msgr2=0x7ff7180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 -- 172.21.15.97:0/453538834 shutdown_connections 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.522+0000 7ff71c901700 1 -- 172.21.15.97:0/453538834 wait complete. 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71c901700 1 Processor -- start 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71c901700 1 -- start start 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71c901700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71c901700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff718106cd0 con 0x7ff7180f5410 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37610/0 (socket says 172.21.15.97:37610) 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.523+0000 7ff71659c700 1 -- 172.21.15.97:0/79294487 learned_addr learned my addr 172.21.15.97:0/79294487 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff71659c700 1 -- 172.21.15.97:0/79294487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708005040 con 0x7ff7180f5410 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff71659c700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ff708005010 tx=0x7ff70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff708009320 con 0x7ff7180f5410 2024-02-19T22:21:35.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff708013380 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff718106ed0 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff708014e50 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.524+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff718107360 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff718045ad0 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff7080095f0 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff706ffd700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 0x7ff7000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff708040f80 con 0x7ff7180f5410 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff715d9b700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 0x7ff7000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.525+0000 7ff715d9b700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 0x7ff7000323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff70c0096a0 tx=0x7ff70c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:35.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.529+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff708014420 con 0x7ff7180f5410 2024-02-19T22:21:35.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.733+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff718108210 con 0x7ff7180f5410 2024-02-19T22:21:35.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.734+0000 7ff706ffd700 1 -- 172.21.15.97:0/79294487 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff708010230 con 0x7ff7180f5410 2024-02-19T22:21:35.736 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:35.736 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:35.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 msgr2=0x7ff7000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 0x7ff7000323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7ff70c0096a0 tx=0x7ff70c006b40).stop 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 msgr2=0x7ff7181067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7ff708005010 tx=0x7ff70800b040).stop 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 shutdown_connections 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff70002ff10 0x7ff7000323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 --2- 172.21.15.97:0/79294487 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff7180f5410 0x7ff7181067e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 >> 172.21.15.97:0/79294487 conn(0x7ff7180f0b70 msgr2=0x7ff7180f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 shutdown_connections 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:35.735+0000 7ff71c901700 1 -- 172.21.15.97:0/79294487 wait complete. 2024-02-19T22:21:35.738 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:21:35.181062+0000 mgr.smithi045.bvlamo (mgr.14162) 232 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi045 ... 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Verifying port 9100 ... 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 1 from systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: systemctl: stderr Job for ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045.service failed because a timeout was exceeded. 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: systemctl: stderr See "systemctl status ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045.service" and "journalctl -xe" for details. 2024-02-19T22:21:36.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ports=daemon_ports) 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:21:36.392 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: Failed command: systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: yield (conn, connr) 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: code, '\n'.join(err))) 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi045 ... 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Verifying port 9100 ... 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 1 from systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: systemctl: stderr Job for ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045.service failed because a timeout was exceeded. 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: systemctl: stderr See "systemctl status ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045.service" and "journalctl -xe" for details. 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:21:36.393 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: ports=daemon_ports) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: Failed command: systemctl start ceph-f6020aec-cf73-11ee-95bb-87774f69a715@node-exporter.smithi045 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:21:35.186705+0000 mgr.smithi045.bvlamo (mgr.14162) 233 : cephadm [INF] Deploying daemon prometheus.smithi045 on smithi045 2024-02-19T22:21:36.394 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:35.734797+0000 mon.smithi045 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.97:0/79294487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:37.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:35.854484+0000 mgr.smithi045.bvlamo (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:37.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:37.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:39.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 -- 172.21.15.97:0/1670102916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 msgr2=0x7fe4980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 --2- 172.21.15.97:0/1670102916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4980f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fe48c009fb0 tx=0x7fe48c009b30).stop 2024-02-19T22:21:39.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 -- 172.21.15.97:0/1670102916 shutdown_connections 2024-02-19T22:21:39.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 --2- 172.21.15.97:0/1670102916 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4980f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 -- 172.21.15.97:0/1670102916 >> 172.21.15.97:0/1670102916 conn(0x7fe4980f0b70 msgr2=0x7fe4980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:39.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 -- 172.21.15.97:0/1670102916 shutdown_connections 2024-02-19T22:21:39.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.521+0000 7fe49d909700 1 -- 172.21.15.97:0/1670102916 wait complete. 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe49d909700 1 Processor -- start 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe49d909700 1 -- start start 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe49d909700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe49d909700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe498105bd0 con 0x7fe4980f5410 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe496ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe496ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54014/0 (socket says 172.21.15.97:54014) 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.522+0000 7fe496ffd700 1 -- 172.21.15.97:0/3455225351 learned_addr learned my addr 172.21.15.97:0/3455225351 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe496ffd700 1 -- 172.21.15.97:0/3455225351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c005040 con 0x7fe4980f5410 2024-02-19T22:21:39.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe496ffd700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe48c005010 tx=0x7fe48c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe48c013790 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe48c008c00 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe498105dd0 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe48c014600 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.523+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe498106260 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.524+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe48c0148c0 con 0x7fe4980f5410 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.524+0000 7fe47f7fe700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:39.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.524+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe48c043590 con 0x7fe4980f5410 2024-02-19T22:21:39.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.524+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe498059100 con 0x7fe4980f5410 2024-02-19T22:21:39.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.524+0000 7fe4967fc700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:39.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.525+0000 7fe4967fc700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 0x7fe480032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe4880096a0 tx=0x7fe488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:39.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.529+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe48c014c30 con 0x7fe4980f5410 2024-02-19T22:21:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.735+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4980fafa0 con 0x7fe4980f5410 2024-02-19T22:21:39.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.735+0000 7fe47f7fe700 1 -- 172.21.15.97:0/3455225351 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe48c02b250 con 0x7fe4980f5410 2024-02-19T22:21:39.738 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:39.738 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 msgr2=0x7fe480032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.739 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 0x7fe480032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fe4880096a0 tx=0x7fe488006b40).stop 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 msgr2=0x7fe4981056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe48c005010 tx=0x7fe48c00b040).stop 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 shutdown_connections 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe48002ff60 0x7fe480032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 --2- 172.21.15.97:0/3455225351 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe4980f5410 0x7fe4981056e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 >> 172.21.15.97:0/3455225351 conn(0x7fe4980f0b70 msgr2=0x7fe4980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 shutdown_connections 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:39.737+0000 7fe49d909700 1 -- 172.21.15.97:0/3455225351 wait complete. 2024-02-19T22:21:39.740 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:39.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:37.854772+0000 mgr.smithi045.bvlamo (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:40.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:39.736696+0000 mon.smithi045 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.97:0/3455225351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:41.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:41.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:39.855102+0000 mgr.smithi045.bvlamo (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- 172.21.15.97:0/2653958226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 msgr2=0x7f50380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 --2- 172.21.15.97:0/2653958226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f5020009fb0 tx=0x7f5020009b30).stop 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- 172.21.15.97:0/2653958226 shutdown_connections 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 --2- 172.21.15.97:0/2653958226 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- 172.21.15.97:0/2653958226 >> 172.21.15.97:0/2653958226 conn(0x7f50380f0b70 msgr2=0x7f50380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- 172.21.15.97:0/2653958226 shutdown_connections 2024-02-19T22:21:43.447 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- 172.21.15.97:0/2653958226 wait complete. 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 Processor -- start 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- start start 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.445+0000 7f503c879700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380fb960 con 0x7f50380f5410 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:54030/0 (socket says 172.21.15.97:54030) 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503659c700 1 -- 172.21.15.97:0/2538482103 learned_addr learned my addr 172.21.15.97:0/2538482103 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503659c700 1 -- 172.21.15.97:0/2538482103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5020005040 con 0x7f50380f5410 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503659c700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f5020005010 tx=0x7f502000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5020009320 con 0x7f50380f5410 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50380fbb00 con 0x7f50380f5410 2024-02-19T22:21:43.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5020013380 con 0x7f50380f5410 2024-02-19T22:21:43.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5020014e50 con 0x7f50380f5410 2024-02-19T22:21:43.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.446+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50380fbf90 con 0x7f50380f5410 2024-02-19T22:21:43.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.447+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50200263a0 con 0x7f50380f5410 2024-02-19T22:21:43.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.447+0000 7f502effd700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 0x7f50240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:43.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.447+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5038059100 con 0x7f50380f5410 2024-02-19T22:21:43.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.447+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5020041290 con 0x7f50380f5410 2024-02-19T22:21:43.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.448+0000 7f5035d9b700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 0x7f50240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:43.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.448+0000 7f5035d9b700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 0x7f50240323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f50280096a0 tx=0x7f5028006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:43.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.453+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5020012920 con 0x7f50380f5410 2024-02-19T22:21:43.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.661+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50380fafa0 con 0x7f50380f5410 2024-02-19T22:21:43.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.662+0000 7f502effd700 1 -- 172.21.15.97:0/2538482103 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5020012330 con 0x7f50380f5410 2024-02-19T22:21:43.665 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:43.665 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:43.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.663+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 msgr2=0x7f50240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.663+0000 7f503c879700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 0x7f50240323d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f50280096a0 tx=0x7f5028006b40).stop 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.663+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 msgr2=0x7f50380fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.663+0000 7f503c879700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f5020005010 tx=0x7f502000b040).stop 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 shutdown_connections 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f502402ff10 0x7f50240323d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 --2- 172.21.15.97:0/2538482103 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f50380f5410 0x7f50380fd4e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:43.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 >> 172.21.15.97:0/2538482103 conn(0x7f50380f0b70 msgr2=0x7f503805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:43.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 shutdown_connections 2024-02-19T22:21:43.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:43.664+0000 7f503c879700 1 -- 172.21.15.97:0/2538482103 wait complete. 2024-02-19T22:21:43.668 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:43.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:41.855490+0000 mgr.smithi045.bvlamo (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:44.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:43.662943+0000 mon.smithi045 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.97:0/2538482103' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:45.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:45.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:45.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:43.855850+0000 mgr.smithi045.bvlamo (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 -- 172.21.15.97:0/2967339196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 msgr2=0x7fd9040f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 --2- 172.21.15.97:0/2967339196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd9040f8f20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fd8f4009fb0 tx=0x7fd8f4009b30).stop 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 -- 172.21.15.97:0/2967339196 shutdown_connections 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 --2- 172.21.15.97:0/2967339196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd9040f8f20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 -- 172.21.15.97:0/2967339196 >> 172.21.15.97:0/2967339196 conn(0x7fd9040f09a0 msgr2=0x7fd9040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 -- 172.21.15.97:0/2967339196 shutdown_connections 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 -- 172.21.15.97:0/2967339196 wait complete. 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.361+0000 7fd908846700 1 Processor -- start 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd908846700 1 -- start start 2024-02-19T22:21:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd908846700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd908846700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd904106af0 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd9037fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd9037fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:57096/0 (socket says 172.21.15.97:57096) 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd9037fe700 1 -- 172.21.15.97:0/3745899135 learned_addr learned my addr 172.21.15.97:0/3745899135 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd9037fe700 1 -- 172.21.15.97:0/3745899135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4005040 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.362+0000 7fd9037fe700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd8f4005010 tx=0x7fd8f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f4009320 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd904106cf0 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8f4013380 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8f4014e50 con 0x7fd9040601e0 2024-02-19T22:21:47.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904107180 con 0x7fd9040601e0 2024-02-19T22:21:47.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.363+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8f4014420 con 0x7fd9040601e0 2024-02-19T22:21:47.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.364+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd904045ad0 con 0x7fd9040601e0 2024-02-19T22:21:47.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.364+0000 7fd8ebfff700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 0x7fd8ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:47.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.364+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8f4040e30 con 0x7fd9040601e0 2024-02-19T22:21:47.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.364+0000 7fd902ffd700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 0x7fd8ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:47.368 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.364+0000 7fd902ffd700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 0x7fd8ec0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd8f80096a0 tx=0x7fd8f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:47.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.368+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8f40095f0 con 0x7fd9040601e0 2024-02-19T22:21:47.580 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.578+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd904108030 con 0x7fd9040601e0 2024-02-19T22:21:47.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.579+0000 7fd8ebfff700 1 -- 172.21.15.97:0/3745899135 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd8f401db70 con 0x7fd9040601e0 2024-02-19T22:21:47.581 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:47.581 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 msgr2=0x7fd8ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 0x7fd8ec0323d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fd8f80096a0 tx=0x7fd8f8006b40).stop 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 msgr2=0x7fd904106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fd8f4005010 tx=0x7fd8f400b040).stop 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 shutdown_connections 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd8ec02ff10 0x7fd8ec0323d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:47.583 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 --2- 172.21.15.97:0/3745899135 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd9040601e0 0x7fd904106600 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:47.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 >> 172.21.15.97:0/3745899135 conn(0x7fd9040f09a0 msgr2=0x7fd9040f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:47.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 shutdown_connections 2024-02-19T22:21:47.584 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:47.581+0000 7fd908846700 1 -- 172.21.15.97:0/3745899135 wait complete. 2024-02-19T22:21:47.584 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:45.856167+0000 mgr.smithi045.bvlamo (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:47.580154+0000 mon.smithi045 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.97:0/3745899135' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:49.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:49.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:47.856496+0000 mgr.smithi045.bvlamo (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- 172.21.15.97:0/3072721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 msgr2=0x7f88480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 --2- 172.21.15.97:0/3072721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f88480f55c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f883c009fb0 tx=0x7f883c009b30).stop 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- 172.21.15.97:0/3072721417 shutdown_connections 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 --2- 172.21.15.97:0/3072721417 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f88480f55c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- 172.21.15.97:0/3072721417 >> 172.21.15.97:0/3072721417 conn(0x7f88480f09a0 msgr2=0x7f88480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- 172.21.15.97:0/3072721417 shutdown_connections 2024-02-19T22:21:51.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- 172.21.15.97:0/3072721417 wait complete. 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 Processor -- start 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.283+0000 7f884dd33700 1 -- start start 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884dd33700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884dd33700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8848106b40 con 0x7f88480f51a0 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884cd31700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884cd31700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:57112/0 (socket says 172.21.15.97:57112) 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884cd31700 1 -- 172.21.15.97:0/40000746 learned_addr learned my addr 172.21.15.97:0/40000746 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884cd31700 1 -- 172.21.15.97:0/40000746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c005040 con 0x7f88480f51a0 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f884cd31700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f883c000c00 tx=0x7f883c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:51.287 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.284+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f883c0095f0 con 0x7f88480f51a0 2024-02-19T22:21:51.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8848106d40 con 0x7f88480f51a0 2024-02-19T22:21:51.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f883c007e90 con 0x7f88480f51a0 2024-02-19T22:21:51.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f883c014db0 con 0x7f88480f51a0 2024-02-19T22:21:51.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88481071d0 con 0x7f88480f51a0 2024-02-19T22:21:51.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f883c025070 con 0x7f88480f51a0 2024-02-19T22:21:51.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.285+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8848045ad0 con 0x7f88480f51a0 2024-02-19T22:21:51.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.286+0000 7f88457fa700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:51.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.286+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f883c040890 con 0x7f88480f51a0 2024-02-19T22:21:51.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.286+0000 7f8847fff700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:51.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.286+0000 7f8847fff700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 0x7f8838032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f88340096a0 tx=0x7f8834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:51.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.291+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f883c00f300 con 0x7f88480f51a0 2024-02-19T22:21:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.499+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8848108000 con 0x7f88480f51a0 2024-02-19T22:21:51.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.499+0000 7f88457fa700 1 -- 172.21.15.97:0/40000746 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f883c00f300 con 0x7f88480f51a0 2024-02-19T22:21:51.501 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:51.501 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 msgr2=0x7f8838032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 0x7f8838032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f88340096a0 tx=0x7f8834006b40).stop 2024-02-19T22:21:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 msgr2=0x7f8848106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f883c000c00 tx=0x7f883c00b040).stop 2024-02-19T22:21:51.503 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 shutdown_connections 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f883802ff60 0x7f8838032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 --2- 172.21.15.97:0/40000746 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88480f51a0 0x7f8848106650 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 >> 172.21.15.97:0/40000746 conn(0x7f88480f09a0 msgr2=0x7f884805fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 shutdown_connections 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:51.501+0000 7f884dd33700 1 -- 172.21.15.97:0/40000746 wait complete. 2024-02-19T22:21:51.504 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:49.856872+0000 mgr.smithi045.bvlamo (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:51.500519+0000 mon.smithi045 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.97:0/40000746' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:53.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:53.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:51.857246+0000 mgr.smithi045.bvlamo (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:55.285 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- 172.21.15.97:0/582105367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 msgr2=0x7fb9640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 --2- 172.21.15.97:0/582105367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640f55c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fb954009fb0 tx=0x7fb954009b30).stop 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- 172.21.15.97:0/582105367 shutdown_connections 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 --2- 172.21.15.97:0/582105367 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640f55c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- 172.21.15.97:0/582105367 >> 172.21.15.97:0/582105367 conn(0x7fb9640f09a0 msgr2=0x7fb9640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- 172.21.15.97:0/582105367 shutdown_connections 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- 172.21.15.97:0/582105367 wait complete. 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 Processor -- start 2024-02-19T22:21:55.286 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.283+0000 7fb96359e700 1 -- start start 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96359e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96359e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9640fb6f0 con 0x7fb9640f51a0 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:57142/0 (socket says 172.21.15.97:57142) 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96259c700 1 -- 172.21.15.97:0/1252079966 learned_addr learned my addr 172.21.15.97:0/1252079966 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96259c700 1 -- 172.21.15.97:0/1252079966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb954005040 con 0x7fb9640f51a0 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96259c700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb954009830 tx=0x7fb95400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb954009320 con 0x7fb9640f51a0 2024-02-19T22:21:55.288 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.284+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9640fb890 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9640fbd20 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb954013380 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb954014420 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb95401e420 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb964059100 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb952ffd700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.285+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb954041160 con 0x7fb9640f51a0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.286+0000 7fb961d9b700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.286+0000 7fb961d9b700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 0x7fb94c032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:55.292 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.290+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb95400fcd0 con 0x7fb9640f51a0 2024-02-19T22:21:55.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.497+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb964045ad0 con 0x7fb9640f51a0 2024-02-19T22:21:55.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.497+0000 7fb952ffd700 1 -- 172.21.15.97:0/1252079966 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb95402c2d0 con 0x7fb9640f51a0 2024-02-19T22:21:55.500 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:55.500 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:55.501 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 msgr2=0x7fb94c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 0x7fb94c032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fb9580096a0 tx=0x7fb958006b40).stop 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 msgr2=0x7fb9640fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fb954009830 tx=0x7fb95400b040).stop 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 shutdown_connections 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb94c02ff60 0x7fb94c032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 --2- 172.21.15.97:0/1252079966 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9640f51a0 0x7fb9640fd270 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 >> 172.21.15.97:0/1252079966 conn(0x7fb9640f09a0 msgr2=0x7fb9640f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 shutdown_connections 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:55.499+0000 7fb96359e700 1 -- 172.21.15.97:0/1252079966 wait complete. 2024-02-19T22:21:55.502 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:53.857634+0000 mgr.smithi045.bvlamo (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:55.498498+0000 mon.smithi045 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.97:0/1252079966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:57.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:57.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:55.858100+0000 mgr.smithi045.bvlamo (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 -- 172.21.15.97:0/2642729888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 msgr2=0x7f18cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 --2- 172.21.15.97:0/2642729888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc0f66c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f18c0009fb0 tx=0x7f18c0009b30).stop 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 -- 172.21.15.97:0/2642729888 shutdown_connections 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 --2- 172.21.15.97:0/2642729888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc0f66c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 -- 172.21.15.97:0/2642729888 >> 172.21.15.97:0/2642729888 conn(0x7f18cc0f1aa0 msgr2=0x7f18cc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 -- 172.21.15.97:0/2642729888 shutdown_connections 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.189+0000 7f18d1f77700 1 -- 172.21.15.97:0/2642729888 wait complete. 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d1f77700 1 Processor -- start 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d1f77700 1 -- start start 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d1f77700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:59.217 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d1f77700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc069e20 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d0f75700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d0f75700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52144/0 (socket says 172.21.15.97:52144) 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.190+0000 7f18d0f75700 1 -- 172.21.15.97:0/3134741911 learned_addr learned my addr 172.21.15.97:0/3134741911 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18d0f75700 1 -- 172.21.15.97:0/3134741911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c0005040 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18d0f75700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f18c00089f0 tx=0x7f18c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c0009320 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc069fc0 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18c0013380 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18c0014e20 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.191+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc06a450 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18c00095f0 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc059100 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18c97fa700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 0x7f18bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18c0040ef0 con 0x7f18cc0f62a0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18cbfff700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 0x7f18bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:59.218 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.192+0000 7f18cbfff700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 0x7f18bc0323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f18b80096a0 tx=0x7f18b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:59.219 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.197+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f18c0014420 con 0x7f18cc0f62a0 2024-02-19T22:21:59.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.401+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18cc045ad0 con 0x7f18cc0f62a0 2024-02-19T22:21:59.403 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.401+0000 7f18c97fa700 1 -- 172.21.15.97:0/3134741911 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f18c0010230 con 0x7f18cc0f62a0 2024-02-19T22:21:59.403 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:21:59.404 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 msgr2=0x7f18bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 0x7f18bc0323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f18b80096a0 tx=0x7f18b8006b40).stop 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 msgr2=0x7f18cc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f18c00089f0 tx=0x7f18c000b040).stop 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 shutdown_connections 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f18bc02ff10 0x7f18bc0323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 --2- 172.21.15.97:0/3134741911 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f18cc0f62a0 0x7f18cc06b9a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:59.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 >> 172.21.15.97:0/3134741911 conn(0x7f18cc0f1aa0 msgr2=0x7f18cc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:59.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 shutdown_connections 2024-02-19T22:21:59.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:21:59.403+0000 7f18d1f77700 1 -- 172.21.15.97:0/3134741911 wait complete. 2024-02-19T22:21:59.406 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:21:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:57.858489+0000 mgr.smithi045.bvlamo (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:21:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:21:59.402412+0000 mon.smithi045 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.97:0/3134741911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:01.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:01.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:21:59.858867+0000 mgr.smithi045.bvlamo (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:03.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- 172.21.15.97:0/1251275326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 msgr2=0x7f88680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:03.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 --2- 172.21.15.97:0/1251275326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88680f66c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f8858009fb0 tx=0x7f8858009b30).stop 2024-02-19T22:22:03.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- 172.21.15.97:0/1251275326 shutdown_connections 2024-02-19T22:22:03.020 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 --2- 172.21.15.97:0/1251275326 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88680f66c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- 172.21.15.97:0/1251275326 >> 172.21.15.97:0/1251275326 conn(0x7f88680f1aa0 msgr2=0x7f88680f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- 172.21.15.97:0/1251275326 shutdown_connections 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- 172.21.15.97:0/1251275326 wait complete. 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 Processor -- start 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.017+0000 7f8867fff700 1 -- start start 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8867fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8867fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8868106ac0 con 0x7f88680f62a0 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8866ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8866ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52174/0 (socket says 172.21.15.97:52174) 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8866ffd700 1 -- 172.21.15.97:0/2745452042 learned_addr learned my addr 172.21.15.97:0/2745452042 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8866ffd700 1 -- 172.21.15.97:0/2745452042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8858005040 con 0x7f88680f62a0 2024-02-19T22:22:03.021 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f8866ffd700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f88580089f0 tx=0x7f885800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:03.022 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.018+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8858009320 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.019+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8868106cc0 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.019+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8858013380 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.019+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8858014e20 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.019+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8868107150 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.019+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88580263a0 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.020+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8868045ad0 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.020+0000 7f884f7fe700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 0x7f88500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.020+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88580411b0 con 0x7f88680f62a0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.020+0000 7f88667fc700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 0x7f88500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:03.023 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.020+0000 7f88667fc700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 0x7f88500323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f885c0096a0 tx=0x7f885c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:03.026 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.025+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f885801cac0 con 0x7f88680f62a0 2024-02-19T22:22:03.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.229+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8868108000 con 0x7f88680f62a0 2024-02-19T22:22:03.232 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.230+0000 7f884f7fe700 1 -- 172.21.15.97:0/2745452042 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f885801cac0 con 0x7f88680f62a0 2024-02-19T22:22:03.232 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:03.232 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 msgr2=0x7f88500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 0x7f88500323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f885c0096a0 tx=0x7f885c006b40).stop 2024-02-19T22:22:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 msgr2=0x7f88681065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f88580089f0 tx=0x7f885800b040).stop 2024-02-19T22:22:03.234 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 shutdown_connections 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f885002ff10 0x7f88500323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 --2- 172.21.15.97:0/2745452042 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f88680f62a0 0x7f88681065d0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 >> 172.21.15.97:0/2745452042 conn(0x7f88680f1aa0 msgr2=0x7f88680f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 shutdown_connections 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:03.232+0000 7f8867fff700 1 -- 172.21.15.97:0/2745452042 wait complete. 2024-02-19T22:22:03.235 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:01.859214+0000 mgr.smithi045.bvlamo (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:03.231060+0000 mon.smithi045 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.97:0/2745452042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:03.937569+0000 mon.smithi045 (mon.0) 254 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:22:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:03.937903+0000 mon.smithi045 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:22:04.925 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:04.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:05.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:03.859613+0000 mgr.smithi045.bvlamo (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:07.013 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- 172.21.15.97:0/3658690325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 msgr2=0x7f29d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 --2- 172.21.15.97:0/3658690325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d40f5830 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f29c8009fb0 tx=0x7f29c8009b30).stop 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- 172.21.15.97:0/3658690325 shutdown_connections 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 --2- 172.21.15.97:0/3658690325 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d40f5830 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- 172.21.15.97:0/3658690325 >> 172.21.15.97:0/3658690325 conn(0x7f29d40f0b70 msgr2=0x7f29d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- 172.21.15.97:0/3658690325 shutdown_connections 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- 172.21.15.97:0/3658690325 wait complete. 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 Processor -- start 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- start start 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.010+0000 7f29dbb68700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d4106cd0 con 0x7f29d40f5410 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29d9904700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29d9904700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:38376/0 (socket says 172.21.15.97:38376) 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29d9904700 1 -- 172.21.15.97:0/1741726370 learned_addr learned my addr 172.21.15.97:0/1741726370 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:07.014 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29d9904700 1 -- 172.21.15.97:0/1741726370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8005040 con 0x7f29d40f5410 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29d9904700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f29c8005010 tx=0x7f29c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c8009320 con 0x7f29d40f5410 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d4106ed0 con 0x7f29d40f5410 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29c8013380 con 0x7f29d40f5410 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c8014e50 con 0x7f29d40f5410 2024-02-19T22:22:07.015 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.011+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d4107360 con 0x7f29d40f5410 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.012+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29c80263a0 con 0x7f29d40f5410 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.012+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4045ad0 con 0x7f29d40f5410 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.012+0000 7f29c67fc700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 0x7f29c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.012+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29c8041140 con 0x7f29d40f5410 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.013+0000 7f29d9103700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 0x7f29c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:07.016 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.013+0000 7f29d9103700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 0x7f29c00323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29d00096a0 tx=0x7f29d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:07.019 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.017+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29c801cb10 con 0x7f29d40f5410 2024-02-19T22:22:07.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.226+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29d4108210 con 0x7f29d40f5410 2024-02-19T22:22:07.228 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.226+0000 7f29c67fc700 1 -- 172.21.15.97:0/1741726370 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f29c801cb10 con 0x7f29d40f5410 2024-02-19T22:22:07.229 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:07.229 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 msgr2=0x7f29c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 0x7f29c00323d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f29d00096a0 tx=0x7f29d0006b40).stop 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 msgr2=0x7f29d41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f29c8005010 tx=0x7f29c800b040).stop 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 shutdown_connections 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f29c002ff10 0x7f29c00323d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 --2- 172.21.15.97:0/1741726370 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f29d40f5410 0x7f29d41067e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 >> 172.21.15.97:0/1741726370 conn(0x7f29d40f0b70 msgr2=0x7f29d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 shutdown_connections 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:07.228+0000 7f29dbb68700 1 -- 172.21.15.97:0/1741726370 wait complete. 2024-02-19T22:22:07.231 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:07.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:05.859964+0000 mgr.smithi045.bvlamo (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:07.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:07.227583+0000 mon.smithi045 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.97:0/1741726370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:08.863 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:08.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:09.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:07.860251+0000 mgr.smithi045.bvlamo (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 -- 172.21.15.97:0/1196716169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 msgr2=0x7f56d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 --2- 172.21.15.97:0/1196716169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d40f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f56c4009fb0 tx=0x7f56c4009b30).stop 2024-02-19T22:22:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 -- 172.21.15.97:0/1196716169 shutdown_connections 2024-02-19T22:22:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 --2- 172.21.15.97:0/1196716169 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d40f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:10.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 -- 172.21.15.97:0/1196716169 >> 172.21.15.97:0/1196716169 conn(0x7f56d40f0b70 msgr2=0x7f56d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 -- 172.21.15.97:0/1196716169 shutdown_connections 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.702+0000 7f56da41e700 1 -- 172.21.15.97:0/1196716169 wait complete. 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56da41e700 1 Processor -- start 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56da41e700 1 -- start start 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56da41e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:10.706 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56da41e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d4060240 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56d3fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:38398/0 (socket says 172.21.15.97:38398) 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.703+0000 7f56d3fff700 1 -- 172.21.15.97:0/1974866554 learned_addr learned my addr 172.21.15.97:0/1974866554 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56d3fff700 1 -- 172.21.15.97:0/1974866554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4005040 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56d3fff700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f56c4005010 tx=0x7f56c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56c4009320 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d4060440 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f56c4013380 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56c4014e50 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.704+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d4109b90 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.705+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56c40095f0 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.705+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d4045ad0 con 0x7f56d40f5410 2024-02-19T22:22:10.708 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.705+0000 7f56d0ff9700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 0x7f56bc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:10.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.705+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f56c4040f60 con 0x7f56d40f5410 2024-02-19T22:22:10.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.705+0000 7f56d37fe700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 0x7f56bc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:10.709 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.706+0000 7f56d37fe700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 0x7f56bc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f56c80096a0 tx=0x7f56c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:10.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.710+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f56c4014420 con 0x7f56d40f5410 2024-02-19T22:22:10.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.918+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56d410a3b0 con 0x7f56d40f5410 2024-02-19T22:22:10.920 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.918+0000 7f56d0ff9700 1 -- 172.21.15.97:0/1974866554 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f56c4010230 con 0x7f56d40f5410 2024-02-19T22:22:10.921 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:10.921 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 msgr2=0x7f56bc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 0x7f56bc0323d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f56c80096a0 tx=0x7f56c8006b40).stop 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 msgr2=0x7f56d41094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f56c4005010 tx=0x7f56c400b040).stop 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 shutdown_connections 2024-02-19T22:22:10.922 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f56bc02ff10 0x7f56bc0323d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:10.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 --2- 172.21.15.97:0/1974866554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f56d40f5410 0x7f56d41094e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:10.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 >> 172.21.15.97:0/1974866554 conn(0x7f56d40f0b70 msgr2=0x7f56d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:10.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 shutdown_connections 2024-02-19T22:22:10.923 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:10.920+0000 7f56da41e700 1 -- 172.21.15.97:0/1974866554 wait complete. 2024-02-19T22:22:10.923 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:09.860546+0000 mgr.smithi045.bvlamo (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:10.919609+0000 mon.smithi045 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.97:0/1974866554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:12.711 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:12.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:11.860858+0000 mgr.smithi045.bvlamo (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:14.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 -- 172.21.15.97:0/1673457740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 msgr2=0x7fee1c0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:14.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 --2- 172.21.15.97:0/1673457740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c0f4b60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fee10009fb0 tx=0x7fee10009b30).stop 2024-02-19T22:22:14.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 -- 172.21.15.97:0/1673457740 shutdown_connections 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 --2- 172.21.15.97:0/1673457740 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c0f4b60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 -- 172.21.15.97:0/1673457740 >> 172.21.15.97:0/1673457740 conn(0x7fee1c0f0260 msgr2=0x7fee1c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 -- 172.21.15.97:0/1673457740 shutdown_connections 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.636+0000 7fee2259f700 1 -- 172.21.15.97:0/1673457740 wait complete. 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.637+0000 7fee2259f700 1 Processor -- start 2024-02-19T22:22:14.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.637+0000 7fee2259f700 1 -- start start 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee2259f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee2259f700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c106ca0 con 0x7fee1c0f4740 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee1bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:38414/0 (socket says 172.21.15.97:38414) 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee1bfff700 1 -- 172.21.15.97:0/4014095187 learned_addr learned my addr 172.21.15.97:0/4014095187 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee1bfff700 1 -- 172.21.15.97:0/4014095187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee10005040 con 0x7fee1c0f4740 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee1bfff700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fee10005010 tx=0x7fee1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee10009320 con 0x7fee1c0f4740 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c106ea0 con 0x7fee1c0f4740 2024-02-19T22:22:14.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee10013380 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee10014e50 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.638+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c107330 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.639+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee100095f0 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.639+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c045ad0 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.639+0000 7fee18ff9700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 0x7fee040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.639+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee10040e90 con 0x7fee1c0f4740 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.640+0000 7fee1b7fe700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 0x7fee040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:14.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.640+0000 7fee1b7fe700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 0x7fee040323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fee0c0096a0 tx=0x7fee0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:14.646 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.645+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee10014420 con 0x7fee1c0f4740 2024-02-19T22:22:14.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.850+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee1c1081e0 con 0x7fee1c0f4740 2024-02-19T22:22:14.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.850+0000 7fee18ff9700 1 -- 172.21.15.97:0/4014095187 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee10010230 con 0x7fee1c0f4740 2024-02-19T22:22:14.853 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:14.853 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:14.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 msgr2=0x7fee040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:14.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 0x7fee040323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fee0c0096a0 tx=0x7fee0c006b40).stop 2024-02-19T22:22:14.854 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 msgr2=0x7fee1c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fee10005010 tx=0x7fee1000b040).stop 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 shutdown_connections 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fee0402ff10 0x7fee040323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 --2- 172.21.15.97:0/4014095187 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fee1c0f4740 0x7fee1c1067b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 >> 172.21.15.97:0/4014095187 conn(0x7fee1c0f0260 msgr2=0x7fee1c0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 shutdown_connections 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:14.852+0000 7fee2259f700 1 -- 172.21.15.97:0/4014095187 wait complete. 2024-02-19T22:22:14.855 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:13.861211+0000 mgr.smithi045.bvlamo (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:14.851541+0000 mon.smithi045 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.97:0/4014095187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:16.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:16.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:17.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:15.861652+0000 mgr.smithi045.bvlamo (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:18.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 -- 172.21.15.97:0/2045729880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 msgr2=0x7f3bbc0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:18.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 --2- 172.21.15.97:0/2045729880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc0fb550 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f3bac009fb0 tx=0x7f3bac009b30).stop 2024-02-19T22:22:18.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 -- 172.21.15.97:0/2045729880 shutdown_connections 2024-02-19T22:22:18.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 --2- 172.21.15.97:0/2045729880 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc0fb550 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 -- 172.21.15.97:0/2045729880 >> 172.21.15.97:0/2045729880 conn(0x7f3bbc0f0b70 msgr2=0x7f3bbc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 -- 172.21.15.97:0/2045729880 shutdown_connections 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.519+0000 7f3bc2181700 1 -- 172.21.15.97:0/2045729880 wait complete. 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bc2181700 1 Processor -- start 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bc2181700 1 -- start start 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bc2181700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bc2181700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc108e90 con 0x7f3bbc060220 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bbb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bbb7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46272/0 (socket says 172.21.15.97:46272) 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.520+0000 7f3bbb7fe700 1 -- 172.21.15.97:0/1315678621 learned_addr learned my addr 172.21.15.97:0/1315678621 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3bbb7fe700 1 -- 172.21.15.97:0/1315678621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac005040 con 0x7f3bbc060220 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3bbb7fe700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3bac0089f0 tx=0x7f3bac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac009320 con 0x7f3bbc060220 2024-02-19T22:22:18.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc109090 con 0x7f3bbc060220 2024-02-19T22:22:18.525 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3bac013380 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bac014e20 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.521+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc109520 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.522+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bac0263a0 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.522+0000 7f3ba3fff700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.522+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bac040f90 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.522+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc045ad0 con 0x7f3bbc060220 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.522+0000 7f3bbaffd700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.523+0000 7f3bbaffd700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3bb00096a0 tx=0x7f3bb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:18.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.527+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bac012920 con 0x7f3bbc060220 2024-02-19T22:22:18.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.733+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bbc10a240 con 0x7f3bbc060220 2024-02-19T22:22:18.735 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.733+0000 7f3ba3fff700 1 -- 172.21.15.97:0/1315678621 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3bac012330 con 0x7f3bbc060220 2024-02-19T22:22:18.736 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:18.736 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 msgr2=0x7f3ba40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 0x7f3ba40323d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3bb00096a0 tx=0x7f3bb0006b40).stop 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 msgr2=0x7f3bbc1089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f3bac0089f0 tx=0x7f3bac00b040).stop 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 shutdown_connections 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3ba402ff10 0x7f3ba40323d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 --2- 172.21.15.97:0/1315678621 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3bbc060220 0x7f3bbc1089a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 >> 172.21.15.97:0/1315678621 conn(0x7f3bbc0f0b70 msgr2=0x7f3bbc0f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:18.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 shutdown_connections 2024-02-19T22:22:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:18.735+0000 7f3bc2181700 1 -- 172.21.15.97:0/1315678621 wait complete. 2024-02-19T22:22:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:19.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:17.861990+0000 mgr.smithi045.bvlamo (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:19.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:18.734360+0000 mon.smithi045 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.97:0/1315678621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:20.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:20.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:21.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:19.862280+0000 mgr.smithi045.bvlamo (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:22.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 -- 172.21.15.97:0/1525718969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 msgr2=0x7f39d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:22.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 --2- 172.21.15.97:0/1525718969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d40f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f39c8009fb0 tx=0x7f39c8009b30).stop 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 -- 172.21.15.97:0/1525718969 shutdown_connections 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 --2- 172.21.15.97:0/1525718969 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d40f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 -- 172.21.15.97:0/1525718969 >> 172.21.15.97:0/1525718969 conn(0x7f39d40f0b70 msgr2=0x7f39d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 -- 172.21.15.97:0/1525718969 shutdown_connections 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.452+0000 7f39dafe2700 1 -- 172.21.15.97:0/1525718969 wait complete. 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39dafe2700 1 Processor -- start 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39dafe2700 1 -- start start 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39dafe2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39dafe2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d41072f0 con 0x7f39d40f5410 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39d8d7e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39d8d7e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46288/0 (socket says 172.21.15.97:46288) 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.453+0000 7f39d8d7e700 1 -- 172.21.15.97:0/3948493080 learned_addr learned my addr 172.21.15.97:0/3948493080 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:22.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39d8d7e700 1 -- 172.21.15.97:0/3948493080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8005040 con 0x7f39d40f5410 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39d8d7e700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f39c8005010 tx=0x7f39c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c8009320 con 0x7f39d40f5410 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d4060220 con 0x7f39d40f5410 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39c8013380 con 0x7f39d40f5410 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c8014e50 con 0x7f39d40f5410 2024-02-19T22:22:22.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.454+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d410bdf0 con 0x7f39d40f5410 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.455+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39c80095f0 con 0x7f39d40f5410 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.455+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4045ad0 con 0x7f39d40f5410 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.455+0000 7f39d17fa700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.455+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39c8040f60 con 0x7f39d40f5410 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.455+0000 7f39d3fff700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:22.459 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.456+0000 7f39d3fff700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 0x7f39c40323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f39c00096a0 tx=0x7f39c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:22.463 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.460+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39c8014420 con 0x7f39d40f5410 2024-02-19T22:22:22.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.666+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39d410c5b0 con 0x7f39d40f5410 2024-02-19T22:22:22.669 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.667+0000 7f39d17fa700 1 -- 172.21.15.97:0/3948493080 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39c8010230 con 0x7f39d40f5410 2024-02-19T22:22:22.669 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:22.669 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:22.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 msgr2=0x7f39c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:22.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 0x7f39c40323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f39c00096a0 tx=0x7f39c0006b40).stop 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 msgr2=0x7f39d410b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f39c8005010 tx=0x7f39c800b040).stop 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 shutdown_connections 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f39c402ff10 0x7f39c40323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 --2- 172.21.15.97:0/3948493080 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f39d40f5410 0x7f39d410b900 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 >> 172.21.15.97:0/3948493080 conn(0x7f39d40f0b70 msgr2=0x7f39d40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 shutdown_connections 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:22.668+0000 7f39dafe2700 1 -- 172.21.15.97:0/3948493080 wait complete. 2024-02-19T22:22:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:23.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:21.862670+0000 mgr.smithi045.bvlamo (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:23.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:22.667961+0000 mon.smithi045 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.97:0/3948493080' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:24.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:24.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:25.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:23.863138+0000 mgr.smithi045.bvlamo (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:26.323 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.312+0000 7fac59aac700 1 -- 172.21.15.97:0/115731498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 msgr2=0x7fac540f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.312+0000 7fac59aac700 1 --2- 172.21.15.97:0/115731498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac540f55c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fac48009fb0 tx=0x7fac48009b30).stop 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- 172.21.15.97:0/115731498 shutdown_connections 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 --2- 172.21.15.97:0/115731498 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac540f55c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- 172.21.15.97:0/115731498 >> 172.21.15.97:0/115731498 conn(0x7fac540f09a0 msgr2=0x7fac540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- 172.21.15.97:0/115731498 shutdown_connections 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- 172.21.15.97:0/115731498 wait complete. 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 Processor -- start 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- start start 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.313+0000 7fac59aac700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac54106aa0 con 0x7fac540f51a0 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac58aaa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:26.324 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac58aaa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46302/0 (socket says 172.21.15.97:46302) 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac58aaa700 1 -- 172.21.15.97:0/3318219996 learned_addr learned my addr 172.21.15.97:0/3318219996 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac58aaa700 1 -- 172.21.15.97:0/3318219996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac48005040 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac58aaa700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fac480089f0 tx=0x7fac4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac48009320 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac54106ca0 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac48013380 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac48014e20 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.314+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac54107130 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.315+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac480095f0 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.315+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac54045ad0 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.315+0000 7fac517fa700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 0x7fac440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.315+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac48040f50 con 0x7fac540f51a0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.315+0000 7fac53fff700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 0x7fac440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:26.325 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.316+0000 7fac53fff700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 0x7fac440323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fac400096a0 tx=0x7fac40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:26.326 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.320+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac48014420 con 0x7fac540f51a0 2024-02-19T22:22:26.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.525+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac54107fe0 con 0x7fac540f51a0 2024-02-19T22:22:26.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.527+0000 7fac517fa700 1 -- 172.21.15.97:0/3318219996 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac4801d980 con 0x7fac540f51a0 2024-02-19T22:22:26.528 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:26.528 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 msgr2=0x7fac440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 0x7fac440323d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fac400096a0 tx=0x7fac40006b40).stop 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 msgr2=0x7fac541065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fac480089f0 tx=0x7fac4800b040).stop 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 shutdown_connections 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac4402ff10 0x7fac440323d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:26.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 --2- 172.21.15.97:0/3318219996 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac540f51a0 0x7fac541065b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:26.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 >> 172.21.15.97:0/3318219996 conn(0x7fac540f09a0 msgr2=0x7fac540f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:26.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 shutdown_connections 2024-02-19T22:22:26.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:26.528+0000 7fac59aac700 1 -- 172.21.15.97:0/3318219996 wait complete. 2024-02-19T22:22:26.531 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:26.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:26.527179+0000 mon.smithi045 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.97:0/3318219996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:25.863570+0000 mgr.smithi045.bvlamo (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:28.105 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:28.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:29.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:27.864057+0000 mgr.smithi045.bvlamo (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:30.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 -- 172.21.15.97:0/693762721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 msgr2=0x7fa9100fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 --2- 172.21.15.97:0/693762721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9100fb570 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fa8f8009fb0 tx=0x7fa8f8009b30).stop 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 -- 172.21.15.97:0/693762721 shutdown_connections 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 --2- 172.21.15.97:0/693762721 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9100fb570 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 -- 172.21.15.97:0/693762721 >> 172.21.15.97:0/693762721 conn(0x7fa9100f0b70 msgr2=0x7fa9100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 -- 172.21.15.97:0/693762721 shutdown_connections 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.251+0000 7fa915cd4700 1 -- 172.21.15.97:0/693762721 wait complete. 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa915cd4700 1 Processor -- start 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa915cd4700 1 -- start start 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa915cd4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa915cd4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa910108e90 con 0x7fa910060200 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa90f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:30.255 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa90f7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48016/0 (socket says 172.21.15.97:48016) 2024-02-19T22:22:30.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.252+0000 7fa90f7fe700 1 -- 172.21.15.97:0/3805458554 learned_addr learned my addr 172.21.15.97:0/3805458554 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:30.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa90f7fe700 1 -- 172.21.15.97:0/3805458554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8005040 con 0x7fa910060200 2024-02-19T22:22:30.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa90f7fe700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa8f80089f0 tx=0x7fa8f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f8009320 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa910109090 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8f8013380 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8f8014e20 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.253+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa910109520 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.254+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8f80263a0 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.254+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa910045ad0 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.254+0000 7fa907fff700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 0x7fa8fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.254+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8f8041160 con 0x7fa910060200 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.254+0000 7fa90effd700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 0x7fa8fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:30.258 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.255+0000 7fa90effd700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 0x7fa8fc0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:30.261 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.259+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8f801cb10 con 0x7fa910060200 2024-02-19T22:22:30.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.470+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa91010a3d0 con 0x7fa910060200 2024-02-19T22:22:30.473 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.471+0000 7fa907fff700 1 -- 172.21.15.97:0/3805458554 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa8f801cb10 con 0x7fa910060200 2024-02-19T22:22:30.473 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:30.473 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:30.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 msgr2=0x7fa8fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:30.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 0x7fa8fc0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fa9000096a0 tx=0x7fa900006b40).stop 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 msgr2=0x7fa9101089a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa8f80089f0 tx=0x7fa8f800b040).stop 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 shutdown_connections 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa8fc02ff10 0x7fa8fc0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 --2- 172.21.15.97:0/3805458554 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa910060200 0x7fa9101089a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 >> 172.21.15.97:0/3805458554 conn(0x7fa9100f0b70 msgr2=0x7fa9100f15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 shutdown_connections 2024-02-19T22:22:30.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:30.472+0000 7fa915cd4700 1 -- 172.21.15.97:0/3805458554 wait complete. 2024-02-19T22:22:30.476 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:30.471693+0000 mon.smithi045 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.97:0/3805458554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:32.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:29.864464+0000 mgr.smithi045.bvlamo (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:32.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:32.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:34.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:31.864916+0000 mgr.smithi045.bvlamo (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:34.243 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.239+0000 7f3828c8d700 1 -- 172.21.15.97:0/3030648312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 msgr2=0x7f38240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.239+0000 7f3828c8d700 1 --2- 172.21.15.97:0/3030648312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38240f55c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f3814009fb0 tx=0x7f3814009b30).stop 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- 172.21.15.97:0/3030648312 shutdown_connections 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 --2- 172.21.15.97:0/3030648312 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38240f55c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- 172.21.15.97:0/3030648312 >> 172.21.15.97:0/3030648312 conn(0x7f38240f09a0 msgr2=0x7f38240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- 172.21.15.97:0/3030648312 shutdown_connections 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- 172.21.15.97:0/3030648312 wait complete. 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 Processor -- start 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- start start 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:34.244 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.240+0000 7f3828c8d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3824106ad0 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f38237fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f38237fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48030/0 (socket says 172.21.15.97:48030) 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f38237fe700 1 -- 172.21.15.97:0/1963903338 learned_addr learned my addr 172.21.15.97:0/1963903338 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f38237fe700 1 -- 172.21.15.97:0/1963903338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3814005040 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f38237fe700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f38140089f0 tx=0x7f381400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814009320 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3824106cd0 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3814013380 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3814014e20 con 0x7f38240f51a0 2024-02-19T22:22:34.245 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.241+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3824107160 con 0x7f38240f51a0 2024-02-19T22:22:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.242+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38140095f0 con 0x7f38240f51a0 2024-02-19T22:22:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.242+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3824045ad0 con 0x7f38240f51a0 2024-02-19T22:22:34.246 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.242+0000 7f380bfff700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 0x7f380c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:34.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.242+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3814040f10 con 0x7f38240f51a0 2024-02-19T22:22:34.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.242+0000 7f3822ffd700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 0x7f380c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:34.247 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.243+0000 7f3822ffd700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 0x7f380c0323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f38180096a0 tx=0x7f3818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:34.249 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.247+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3814014420 con 0x7f38240f51a0 2024-02-19T22:22:34.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.456+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3824108010 con 0x7f38240f51a0 2024-02-19T22:22:34.460 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.458+0000 7f380bfff700 1 -- 172.21.15.97:0/1963903338 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f381401d980 con 0x7f38240f51a0 2024-02-19T22:22:34.460 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:34.460 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 msgr2=0x7f380c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 0x7f380c0323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f38180096a0 tx=0x7f3818006b40).stop 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 msgr2=0x7f38241065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f38140089f0 tx=0x7f381400b040).stop 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 shutdown_connections 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f380c02ff10 0x7f380c0323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 --2- 172.21.15.97:0/1963903338 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f38240f51a0 0x7f38241065e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 >> 172.21.15.97:0/1963903338 conn(0x7f38240f09a0 msgr2=0x7f38240f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 shutdown_connections 2024-02-19T22:22:34.462 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:34.459+0000 7f3828c8d700 1 -- 172.21.15.97:0/1963903338 wait complete. 2024-02-19T22:22:34.463 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:35.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:34.457782+0000 mon.smithi045 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.97:0/1963903338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:36.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:33.865348+0000 mgr.smithi045.bvlamo (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:36.214 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:36.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:38.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:35.865771+0000 mgr.smithi045.bvlamo (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:38.197 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 -- 172.21.15.97:0/3759129615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 msgr2=0x7f15ec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 --2- 172.21.15.97:0/3759129615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec0f66c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f15dc009fb0 tx=0x7f15dc009b30).stop 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 -- 172.21.15.97:0/3759129615 shutdown_connections 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 --2- 172.21.15.97:0/3759129615 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec0f66c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 -- 172.21.15.97:0/3759129615 >> 172.21.15.97:0/3759129615 conn(0x7f15ec0f1aa0 msgr2=0x7f15ec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 -- 172.21.15.97:0/3759129615 shutdown_connections 2024-02-19T22:22:38.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.195+0000 7f15f12d4700 1 -- 172.21.15.97:0/3759129615 wait complete. 2024-02-19T22:22:38.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15f12d4700 1 Processor -- start 2024-02-19T22:22:38.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15f12d4700 1 -- start start 2024-02-19T22:22:38.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15f12d4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15f12d4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec106ac0 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15ebfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15ebfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46814/0 (socket says 172.21.15.97:46814) 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.196+0000 7f15ebfff700 1 -- 172.21.15.97:0/1391892064 learned_addr learned my addr 172.21.15.97:0/1391892064 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15ebfff700 1 -- 172.21.15.97:0/1391892064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc005040 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15ebfff700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f15dc009830 tx=0x7f15dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15dc009320 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec106cc0 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec107150 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15dc013380 con 0x7f15ec0f62a0 2024-02-19T22:22:38.200 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.197+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15dc014420 con 0x7f15ec0f62a0 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.198+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15cc004fa0 con 0x7f15ec0f62a0 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.198+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15dc01e420 con 0x7f15ec0f62a0 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.198+0000 7f15e8ff9700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 0x7f15d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.198+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15dc040e20 con 0x7f15ec0f62a0 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.198+0000 7f15eb7fe700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 0x7f15d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:38.201 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.199+0000 7f15eb7fe700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 0x7f15d4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:38.205 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.203+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15dc0095f0 con 0x7f15ec0f62a0 2024-02-19T22:22:38.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.406+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15cc005670 con 0x7f15ec0f62a0 2024-02-19T22:22:38.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.406+0000 7f15e8ff9700 1 -- 172.21.15.97:0/1391892064 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15dc027db0 con 0x7f15ec0f62a0 2024-02-19T22:22:38.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:38.409 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:38.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 msgr2=0x7f15d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 0x7f15d4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).stop 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 msgr2=0x7f15ec1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f15dc009830 tx=0x7f15dc00b040).stop 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 shutdown_connections 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f15d402ff60 0x7f15d4032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 --2- 172.21.15.97:0/1391892064 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f15ec0f62a0 0x7f15ec1065d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.408+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 >> 172.21.15.97:0/1391892064 conn(0x7f15ec0f1aa0 msgr2=0x7f15ec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.409+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 shutdown_connections 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:38.409+0000 7f15f12d4700 1 -- 172.21.15.97:0/1391892064 wait complete. 2024-02-19T22:22:38.411 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:39.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:37.866176+0000 mgr.smithi045.bvlamo (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:39.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:38.407609+0000 mon.smithi045 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.97:0/1391892064' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:40.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:40.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:41.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:39.866584+0000 mgr.smithi045.bvlamo (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:42.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 -- 172.21.15.97:0/3804353691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 msgr2=0x7ffb400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:42.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 --2- 172.21.15.97:0/3804353691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb400f55c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7ffb30009fb0 tx=0x7ffb30009b30).stop 2024-02-19T22:22:42.041 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 -- 172.21.15.97:0/3804353691 shutdown_connections 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 --2- 172.21.15.97:0/3804353691 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb400f55c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 -- 172.21.15.97:0/3804353691 >> 172.21.15.97:0/3804353691 conn(0x7ffb400f09a0 msgr2=0x7ffb400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 -- 172.21.15.97:0/3804353691 shutdown_connections 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.037+0000 7ffb449cf700 1 -- 172.21.15.97:0/3804353691 wait complete. 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb449cf700 1 Processor -- start 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb449cf700 1 -- start start 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb449cf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb449cf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb40060220 con 0x7ffb400f51a0 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb3ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb3ed9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46832/0 (socket says 172.21.15.97:46832) 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.038+0000 7ffb3ed9d700 1 -- 172.21.15.97:0/2405740766 learned_addr learned my addr 172.21.15.97:0/2405740766 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb3ed9d700 1 -- 172.21.15.97:0/2405740766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb30005040 con 0x7ffb400f51a0 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb3ed9d700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ffb300089f0 tx=0x7ffb3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:42.042 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb30009320 con 0x7ffb400f51a0 2024-02-19T22:22:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb40060420 con 0x7ffb400f51a0 2024-02-19T22:22:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffb30013380 con 0x7ffb400f51a0 2024-02-19T22:22:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb30014e20 con 0x7ffb400f51a0 2024-02-19T22:22:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.039+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb401099a0 con 0x7ffb400f51a0 2024-02-19T22:22:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.040+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb40059100 con 0x7ffb400f51a0 2024-02-19T22:22:42.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.040+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb300095f0 con 0x7ffb400f51a0 2024-02-19T22:22:42.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.040+0000 7ffb277fe700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 0x7ffb280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:42.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.040+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb30040fb0 con 0x7ffb400f51a0 2024-02-19T22:22:42.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.040+0000 7ffb3e59c700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 0x7ffb280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:42.044 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.041+0000 7ffb3e59c700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 0x7ffb280323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ffb340096a0 tx=0x7ffb34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:42.046 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.045+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb30014420 con 0x7ffb400f51a0 2024-02-19T22:22:42.253 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.251+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb400fad30 con 0x7ffb400f51a0 2024-02-19T22:22:42.254 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.252+0000 7ffb277fe700 1 -- 172.21.15.97:0/2405740766 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ffb30010230 con 0x7ffb400f51a0 2024-02-19T22:22:42.254 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:42.254 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 msgr2=0x7ffb280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 0x7ffb280323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ffb340096a0 tx=0x7ffb34006b40).stop 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 msgr2=0x7ffb401092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ffb300089f0 tx=0x7ffb3000b040).stop 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 shutdown_connections 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ffb2802ff10 0x7ffb280323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 --2- 172.21.15.97:0/2405740766 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ffb400f51a0 0x7ffb401092f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:42.256 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 >> 172.21.15.97:0/2405740766 conn(0x7ffb400f09a0 msgr2=0x7ffb400f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 shutdown_connections 2024-02-19T22:22:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:42.254+0000 7ffb449cf700 1 -- 172.21.15.97:0/2405740766 wait complete. 2024-02-19T22:22:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:43.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:41.866989+0000 mgr.smithi045.bvlamo (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:43.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:42.253236+0000 mon.smithi045 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.97:0/2405740766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:44.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:44.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:45.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:43.867324+0000 mgr.smithi045.bvlamo (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:46.100 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.097+0000 7ff8ff7da700 1 -- 172.21.15.97:0/2553481444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 msgr2=0x7ff8f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.097+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/2553481444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f80f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff8ec009fb0 tx=0x7ff8ec009b30).stop 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.097+0000 7ff8ff7da700 1 -- 172.21.15.97:0/2553481444 shutdown_connections 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.097+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/2553481444 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f80f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.097+0000 7ff8ff7da700 1 -- 172.21.15.97:0/2553481444 >> 172.21.15.97:0/2553481444 conn(0x7ff8f80f1c70 msgr2=0x7ff8f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 -- 172.21.15.97:0/2553481444 shutdown_connections 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 -- 172.21.15.97:0/2553481444 wait complete. 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 Processor -- start 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 -- start start 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8ff7da700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f8069e20 con 0x7ff8f80f6510 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8fd576700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:46.101 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8fd576700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46844/0 (socket says 172.21.15.97:46844) 2024-02-19T22:22:46.102 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.098+0000 7ff8fd576700 1 -- 172.21.15.97:0/3605682521 learned_addr learned my addr 172.21.15.97:0/3605682521 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8fd576700 1 -- 172.21.15.97:0/3605682521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec005040 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8fd576700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff8ec005010 tx=0x7ff8ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec009320 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f8069fc0 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f806a450 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff8ec013380 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.099+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8ec014df0 con 0x7ff8f80f6510 2024-02-19T22:22:46.103 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.100+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff8ec0095f0 con 0x7ff8f80f6510 2024-02-19T22:22:46.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.100+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d8004fa0 con 0x7ff8f80f6510 2024-02-19T22:22:46.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.101+0000 7ff8e9ffb700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 0x7ff8e40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:46.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.101+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8ec040df0 con 0x7ff8f80f6510 2024-02-19T22:22:46.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.101+0000 7ff8fcd75700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 0x7ff8e40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:46.104 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.101+0000 7ff8fcd75700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 0x7ff8e40323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:46.107 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.106+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8ec014420 con 0x7ff8f80f6510 2024-02-19T22:22:46.315 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.313+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8d8005670 con 0x7ff8f80f6510 2024-02-19T22:22:46.316 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.314+0000 7ff8e9ffb700 1 -- 172.21.15.97:0/3605682521 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff8ec01d9d0 con 0x7ff8f80f6510 2024-02-19T22:22:46.316 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:46.316 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 msgr2=0x7ff8e40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 0x7ff8e40323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ff8f40096a0 tx=0x7ff8f4006b40).stop 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 msgr2=0x7ff8f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ff8ec005010 tx=0x7ff8ec00b040).stop 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 shutdown_connections 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7ff8e402ff10 0x7ff8e40323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 --2- 172.21.15.97:0/3605682521 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7ff8f80f6510 0x7ff8f806b9a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 >> 172.21.15.97:0/3605682521 conn(0x7ff8f80f1c70 msgr2=0x7ff8f80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:46.318 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 shutdown_connections 2024-02-19T22:22:46.319 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:46.316+0000 7ff8ff7da700 1 -- 172.21.15.97:0/3605682521 wait complete. 2024-02-19T22:22:46.319 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:47.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:45.867634+0000 mgr.smithi045.bvlamo (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:47.391 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:46.314837+0000 mon.smithi045 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.97:0/3605682521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:47.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:47.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:47.868044+0000 mgr.smithi045.bvlamo (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:50.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 -- 172.21.15.97:0/645270411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 msgr2=0x7f800c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:50.114 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 --2- 172.21.15.97:0/645270411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c0f5830 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f7ffc009fb0 tx=0x7f7ffc009b30).stop 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 -- 172.21.15.97:0/645270411 shutdown_connections 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 --2- 172.21.15.97:0/645270411 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c0f5830 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 -- 172.21.15.97:0/645270411 >> 172.21.15.97:0/645270411 conn(0x7f800c0f0b70 msgr2=0x7f800c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 -- 172.21.15.97:0/645270411 shutdown_connections 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.111+0000 7f8012a36700 1 -- 172.21.15.97:0/645270411 wait complete. 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.112+0000 7f8012a36700 1 Processor -- start 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.112+0000 7f8012a36700 1 -- start start 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.112+0000 7f8012a36700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:50.115 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.112+0000 7f8012a36700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c106ca0 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f800bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f800bfff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42450/0 (socket says 172.21.15.97:42450) 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f800bfff700 1 -- 172.21.15.97:0/1062828251 learned_addr learned my addr 172.21.15.97:0/1062828251 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f800bfff700 1 -- 172.21.15.97:0/1062828251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc005040 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f800bfff700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7ffc005010 tx=0x7f7ffc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ffc009320 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c106ea0 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ffc013380 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ffc014e50 con 0x7f800c0f5410 2024-02-19T22:22:50.117 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.113+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c107330 con 0x7f800c0f5410 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.114+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7ffc014420 con 0x7f800c0f5410 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.114+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c045ad0 con 0x7f800c0f5410 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.114+0000 7f8008ff9700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 0x7f7ff40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.114+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7ffc040f40 con 0x7f800c0f5410 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.115+0000 7f800b7fe700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 0x7f7ff40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:50.118 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.115+0000 7f800b7fe700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 0x7f7ff40323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f80000096a0 tx=0x7f8000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:50.121 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.119+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7ffc0095f0 con 0x7f800c0f5410 2024-02-19T22:22:50.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.325+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f800c0fafa0 con 0x7f800c0f5410 2024-02-19T22:22:50.327 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.325+0000 7f8008ff9700 1 -- 172.21.15.97:0/1062828251 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7ffc01db70 con 0x7f800c0f5410 2024-02-19T22:22:50.328 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:50.328 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 msgr2=0x7f7ff40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 0x7f7ff40323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f80000096a0 tx=0x7f8000006b40).stop 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 msgr2=0x7f800c1067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f7ffc005010 tx=0x7f7ffc00b040).stop 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 shutdown_connections 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7ff402ff10 0x7f7ff40323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 --2- 172.21.15.97:0/1062828251 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f800c0f5410 0x7f800c1067b0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:50.329 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 >> 172.21.15.97:0/1062828251 conn(0x7f800c0f0b70 msgr2=0x7f800c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:50.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 shutdown_connections 2024-02-19T22:22:50.330 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:50.327+0000 7f8012a36700 1 -- 172.21.15.97:0/1062828251 wait complete. 2024-02-19T22:22:50.330 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:50.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:50.326469+0000 mon.smithi045 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.97:0/1062828251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:49.868450+0000 mgr.smithi045.bvlamo (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:52.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:52.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- 172.21.15.97:0/1714063101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 msgr2=0x7f70cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 --2- 172.21.15.97:0/1714063101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc0f55c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f70c0009fb0 tx=0x7f70c0009b30).stop 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- 172.21.15.97:0/1714063101 shutdown_connections 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 --2- 172.21.15.97:0/1714063101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc0f55c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- 172.21.15.97:0/1714063101 >> 172.21.15.97:0/1714063101 conn(0x7f70cc0f09a0 msgr2=0x7f70cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- 172.21.15.97:0/1714063101 shutdown_connections 2024-02-19T22:22:53.858 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- 172.21.15.97:0/1714063101 wait complete. 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 Processor -- start 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.856+0000 7f70cb59e700 1 -- start start 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70cb59e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70cb59e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc1059c0 con 0x7f70cc0f51a0 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70ca59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70ca59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42470/0 (socket says 172.21.15.97:42470) 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70ca59c700 1 -- 172.21.15.97:0/3064259052 learned_addr learned my addr 172.21.15.97:0/3064259052 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:53.860 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70ca59c700 1 -- 172.21.15.97:0/3064259052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c0005040 con 0x7f70cc0f51a0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70ca59c700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f70c00089f0 tx=0x7f70c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70c0009320 con 0x7f70cc0f51a0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70cc105bc0 con 0x7f70cc0f51a0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.857+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f70c0013380 con 0x7f70cc0f51a0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.858+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70c0014e20 con 0x7f70cc0f51a0 2024-02-19T22:22:53.861 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.858+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc106050 con 0x7f70cc0f51a0 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.858+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc045ad0 con 0x7f70cc0f51a0 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.858+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70c00095f0 con 0x7f70cc0f51a0 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.858+0000 7f70baffd700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 0x7f70b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.859+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70c0040ef0 con 0x7f70cc0f51a0 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.859+0000 7f70c9d9b700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 0x7f70b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:53.862 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.859+0000 7f70c9d9b700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 0x7f70b40323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f70bc0096a0 tx=0x7f70bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:53.865 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:53.863+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70c0014420 con 0x7f70cc0f51a0 2024-02-19T22:22:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:51.868937+0000 mgr.smithi045.bvlamo (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:54.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.067+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70cc106ea0 con 0x7f70cc0f51a0 2024-02-19T22:22:54.070 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.068+0000 7f70baffd700 1 -- 172.21.15.97:0/3064259052 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f70c0010230 con 0x7f70cc0f51a0 2024-02-19T22:22:54.070 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:54.070 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 msgr2=0x7f70b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 0x7f70b40323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f70bc0096a0 tx=0x7f70bc006b40).stop 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 msgr2=0x7f70cc1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f70c00089f0 tx=0x7f70c000b040).stop 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 shutdown_connections 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f70b402ff10 0x7f70b40323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:54.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 --2- 172.21.15.97:0/3064259052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70cc0f51a0 0x7f70cc1054d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:54.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.069+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 >> 172.21.15.97:0/3064259052 conn(0x7f70cc0f09a0 msgr2=0x7f70cc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:54.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.070+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 shutdown_connections 2024-02-19T22:22:54.073 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:54.070+0000 7f70cb59e700 1 -- 172.21.15.97:0/3064259052 wait complete. 2024-02-19T22:22:54.073 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:54.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:54.069162+0000 mon.smithi045 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.97:0/3064259052' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:55.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:55.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:53.869201+0000 mgr.smithi045.bvlamo (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/3537837893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 msgr2=0x7f5cd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/3537837893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd80f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f5cc0009fb0 tx=0x7f5cc0009b30).stop 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/3537837893 shutdown_connections 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/3537837893 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd80f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/3537837893 >> 172.21.15.97:0/3537837893 conn(0x7f5cd80f0b70 msgr2=0x7f5cd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:57.849 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/3537837893 shutdown_connections 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.846+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/3537837893 wait complete. 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cdd1ca700 1 Processor -- start 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cdd1ca700 1 -- start start 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cdd1ca700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cdd1ca700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cd8060240 con 0x7f5cd80f5410 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cd6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cd6d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52078/0 (socket says 172.21.15.97:52078) 2024-02-19T22:22:57.850 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cd6d9d700 1 -- 172.21.15.97:0/1460251641 learned_addr learned my addr 172.21.15.97:0/1460251641 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:22:57.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cd6d9d700 1 -- 172.21.15.97:0/1460251641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc0005040 con 0x7f5cd80f5410 2024-02-19T22:22:57.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.847+0000 7f5cd6d9d700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f5cc0005010 tx=0x7f5cc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:57.851 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc0009320 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cd8060440 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5cc0013380 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5cc0014e50 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cd8109b90 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5cc00095f0 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.848+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cd8045ad0 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.849+0000 7f5ccf7fe700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 0x7f5cc40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.849+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5cc0040fa0 con 0x7f5cd80f5410 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.849+0000 7f5cd659c700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 0x7f5cc40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:57.852 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.849+0000 7f5cd659c700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 0x7f5cc40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5cc80096a0 tx=0x7f5cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:57.856 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:57.854+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5cc0014420 con 0x7f5cd80f5410 2024-02-19T22:22:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:55.869630+0000 mgr.smithi045.bvlamo (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:58.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.067+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5cd810a3b0 con 0x7f5cd80f5410 2024-02-19T22:22:58.069 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.067+0000 7f5ccf7fe700 1 -- 172.21.15.97:0/1460251641 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5cc0010230 con 0x7f5cd80f5410 2024-02-19T22:22:58.070 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:22:58.070 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:58.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 msgr2=0x7f5cc40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:58.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 0x7f5cc40323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f5cc80096a0 tx=0x7f5cc8006b40).stop 2024-02-19T22:22:58.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 msgr2=0x7f5cd81094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:58.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f5cc0005010 tx=0x7f5cc000b040).stop 2024-02-19T22:22:58.071 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 shutdown_connections 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5cc402ff10 0x7f5cc40323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 --2- 172.21.15.97:0/1460251641 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5cd80f5410 0x7f5cd81094e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 >> 172.21.15.97:0/1460251641 conn(0x7f5cd80f0b70 msgr2=0x7f5cd80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 shutdown_connections 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:22:58.069+0000 7f5cdd1ca700 1 -- 172.21.15.97:0/1460251641 wait complete. 2024-02-19T22:22:58.072 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:22:58.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:22:58.068586+0000 mon.smithi045 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.97:0/1460251641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:59.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:59.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:22:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:57.870054+0000 mgr.smithi045.bvlamo (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:01.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.825+0000 7f09fdf97700 1 -- 172.21.15.97:0/1780581104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f80f6510 msgr2=0x7f09f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:01.829 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.825+0000 7f09fdf97700 1 --2- 172.21.15.97:0/1780581104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f80f6510 0x7f09f80f6930 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f09ec009fb0 tx=0x7f09ec009b30).stop 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- 172.21.15.97:0/1780581104 shutdown_connections 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 --2- 172.21.15.97:0/1780581104 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f80f6510 0x7f09f80f6930 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- 172.21.15.97:0/1780581104 >> 172.21.15.97:0/1780581104 conn(0x7f09f80f1c70 msgr2=0x7f09f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- 172.21.15.97:0/1780581104 shutdown_connections 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- 172.21.15.97:0/1780581104 wait complete. 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 Processor -- start 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- start start 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.826+0000 7f09fdf97700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8109400 con 0x7f09f8108af0 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09f77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09f77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52096/0 (socket says 172.21.15.97:52096) 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09f77fe700 1 -- 172.21.15.97:0/2060379090 learned_addr learned my addr 172.21.15.97:0/2060379090 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09f77fe700 1 -- 172.21.15.97:0/2060379090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec005040 con 0x7f09f8108af0 2024-02-19T22:23:01.830 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09f77fe700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f09f80f7610 tx=0x7f09ec009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:01.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec013380 con 0x7f09f8108af0 2024-02-19T22:23:01.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09ec013790 con 0x7f09f8108af0 2024-02-19T22:23:01.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8109600 con 0x7f09f8108af0 2024-02-19T22:23:01.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09ec01ee70 con 0x7f09f8108af0 2024-02-19T22:23:01.831 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.827+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f810a150 con 0x7f09f8108af0 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.828+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09ec025060 con 0x7f09f8108af0 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.828+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8059100 con 0x7f09f8108af0 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.828+0000 7f09dffff700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 0x7f09e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.828+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f09ec040dd0 con 0x7f09f8108af0 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.829+0000 7f09f6ffd700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 0x7f09e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:01.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.829+0000 7f09f6ffd700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 0x7f09e0032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f09e80096a0 tx=0x7f09e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:01.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:01.833+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f09ec01c950 con 0x7f09f8108af0 2024-02-19T22:23:01.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:22:59.870390+0000 mgr.smithi045.bvlamo (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:02.036 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.034+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f8045ad0 con 0x7f09f8108af0 2024-02-19T22:23:02.037 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.035+0000 7f09dffff700 1 -- 172.21.15.97:0/2060379090 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f09ec02d2d0 con 0x7f09f8108af0 2024-02-19T22:23:02.037 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:02.037 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 msgr2=0x7f09e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 0x7f09e0032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f09e80096a0 tx=0x7f09e8006b40).stop 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 msgr2=0x7f09f8108f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f09f80f7610 tx=0x7f09ec009b30).stop 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 shutdown_connections 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f09e002ff60 0x7f09e0032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 --2- 172.21.15.97:0/2060379090 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f09f8108af0 0x7f09f8108f10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 >> 172.21.15.97:0/2060379090 conn(0x7f09f80f1c70 msgr2=0x7f09f80f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 shutdown_connections 2024-02-19T22:23:02.039 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:02.036+0000 7f09fdf97700 1 -- 172.21.15.97:0/2060379090 wait complete. 2024-02-19T22:23:02.040 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:02.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:02.036014+0000 mon.smithi045 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.97:0/2060379090' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:03.778 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:03.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:01.870767+0000 mgr.smithi045.bvlamo (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:03.938739+0000 mon.smithi045 (mon.0) 271 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:23:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:03.938965+0000 mon.smithi045 (mon.0) 272 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:23:05.876 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 -- 172.21.15.97:0/2115145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 msgr2=0x7fcd140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 --2- 172.21.15.97:0/2115145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fcd04009fb0 tx=0x7fcd04009b30).stop 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 -- 172.21.15.97:0/2115145935 shutdown_connections 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 --2- 172.21.15.97:0/2115145935 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 -- 172.21.15.97:0/2115145935 >> 172.21.15.97:0/2115145935 conn(0x7fcd140f0b70 msgr2=0x7fcd140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 -- 172.21.15.97:0/2115145935 shutdown_connections 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.873+0000 7fcd1c232700 1 -- 172.21.15.97:0/2115145935 wait complete. 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd1c232700 1 Processor -- start 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd1c232700 1 -- start start 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd1c232700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd1c232700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd140fb960 con 0x7fcd140f5410 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd19fce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd19fce700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:52114/0 (socket says 172.21.15.97:52114) 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd19fce700 1 -- 172.21.15.97:0/320268389 learned_addr learned my addr 172.21.15.97:0/320268389 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:05.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.874+0000 7fcd19fce700 1 -- 172.21.15.97:0/320268389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd04005040 con 0x7fcd140f5410 2024-02-19T22:23:05.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd19fce700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fcd04005010 tx=0x7fcd0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:05.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd04009320 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd140fbb00 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd04013380 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd04014e50 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.875+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd140fbf90 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd040095f0 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd14059100 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd0a7fc700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 0x7fcd000323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd04040fa0 con 0x7fcd140f5410 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd197cd700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 0x7fcd000323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:05.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.876+0000 7fcd197cd700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 0x7fcd000323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd100096a0 tx=0x7fcd10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:05.883 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:05.881+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd04014420 con 0x7fcd140f5410 2024-02-19T22:23:05.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:03.871206+0000 mgr.smithi045.bvlamo (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:06.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.090+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd140fafa0 con 0x7fcd140f5410 2024-02-19T22:23:06.093 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.090+0000 7fcd0a7fc700 1 -- 172.21.15.97:0/320268389 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd04010230 con 0x7fcd140f5410 2024-02-19T22:23:06.093 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:06.093 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 msgr2=0x7fcd000323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 0x7fcd000323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcd100096a0 tx=0x7fcd10006b40).stop 2024-02-19T22:23:06.094 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 msgr2=0x7fcd140fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fcd04005010 tx=0x7fcd0400b040).stop 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 shutdown_connections 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fcd0002ff10 0x7fcd000323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 --2- 172.21.15.97:0/320268389 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fcd140f5410 0x7fcd140fd4e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 >> 172.21.15.97:0/320268389 conn(0x7fcd140f0b70 msgr2=0x7fcd1405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 shutdown_connections 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:06.092+0000 7fcd1c232700 1 -- 172.21.15.97:0/320268389 wait complete. 2024-02-19T22:23:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:06.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:06.091576+0000 mon.smithi045 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.97:0/320268389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:07.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:07.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:07.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:05.871678+0000 mgr.smithi045.bvlamo (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.872+0000 7fa025913700 1 -- 172.21.15.97:0/285858021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 msgr2=0x7fa0200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.872+0000 7fa025913700 1 --2- 172.21.15.97:0/285858021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0200f55c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fa014009fb0 tx=0x7fa014009b30).stop 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- 172.21.15.97:0/285858021 shutdown_connections 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 --2- 172.21.15.97:0/285858021 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0200f55c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- 172.21.15.97:0/285858021 >> 172.21.15.97:0/285858021 conn(0x7fa0200f09a0 msgr2=0x7fa0200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- 172.21.15.97:0/285858021 shutdown_connections 2024-02-19T22:23:09.875 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- 172.21.15.97:0/285858021 wait complete. 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 Processor -- start 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- start start 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.873+0000 7fa025913700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0201059c0 con 0x7fa0200f51a0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa024911700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa024911700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46436/0 (socket says 172.21.15.97:46436) 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa024911700 1 -- 172.21.15.97:0/3180992508 learned_addr learned my addr 172.21.15.97:0/3180992508 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa024911700 1 -- 172.21.15.97:0/3180992508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa014005040 con 0x7fa0200f51a0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa024911700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa0140089f0 tx=0x7fa01400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa014009320 con 0x7fa0200f51a0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa020105bc0 con 0x7fa0200f51a0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa014013380 con 0x7fa0200f51a0 2024-02-19T22:23:09.877 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa014014e20 con 0x7fa0200f51a0 2024-02-19T22:23:09.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.874+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa020106050 con 0x7fa0200f51a0 2024-02-19T22:23:09.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.875+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0140095f0 con 0x7fa0200f51a0 2024-02-19T22:23:09.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.875+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa020045ad0 con 0x7fa0200f51a0 2024-02-19T22:23:09.878 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.875+0000 7fa01d7fa700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 0x7fa0100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:09.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.875+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa014040f50 con 0x7fa0200f51a0 2024-02-19T22:23:09.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.876+0000 7fa01ffff700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 0x7fa0100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:09.879 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.876+0000 7fa01ffff700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 0x7fa0100323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa00c0096a0 tx=0x7fa00c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:09.882 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:09.880+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa014014420 con 0x7fa0200f51a0 2024-02-19T22:23:09.890 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:07.872154+0000 mgr.smithi045.bvlamo (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:10.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.086+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa020106f00 con 0x7fa0200f51a0 2024-02-19T22:23:10.088 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.086+0000 7fa01d7fa700 1 -- 172.21.15.97:0/3180992508 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa01401d980 con 0x7fa0200f51a0 2024-02-19T22:23:10.089 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:10.089 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:10.090 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 msgr2=0x7fa0100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 0x7fa0100323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fa00c0096a0 tx=0x7fa00c006b40).stop 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 msgr2=0x7fa0201054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fa0140089f0 tx=0x7fa01400b040).stop 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 shutdown_connections 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa01002ff10 0x7fa0100323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:10.091 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 --2- 172.21.15.97:0/3180992508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0200f51a0 0x7fa0201054d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:10.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 >> 172.21.15.97:0/3180992508 conn(0x7fa0200f09a0 msgr2=0x7fa0200f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:10.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 shutdown_connections 2024-02-19T22:23:10.092 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:10.088+0000 7fa025913700 1 -- 172.21.15.97:0/3180992508 wait complete. 2024-02-19T22:23:10.093 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:10.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:10.087594+0000 mon.smithi045 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.97:0/3180992508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:11.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:11.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:11.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:09.872475+0000 mgr.smithi045.bvlamo (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 -- 172.21.15.97:0/800982896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 msgr2=0x7fa0c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/800982896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c80f55c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fa0c4009fb0 tx=0x7fa0c4009b30).stop 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 -- 172.21.15.97:0/800982896 shutdown_connections 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/800982896 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c80f55c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 -- 172.21.15.97:0/800982896 >> 172.21.15.97:0/800982896 conn(0x7fa0c80f09a0 msgr2=0x7fa0c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 -- 172.21.15.97:0/800982896 shutdown_connections 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 -- 172.21.15.97:0/800982896 wait complete. 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.780+0000 7fa0cec3d700 1 Processor -- start 2024-02-19T22:23:13.783 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cec3d700 1 -- start start 2024-02-19T22:23:13.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cec3d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:13.784 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cec3d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8104880 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cdc3b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cdc3b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46442/0 (socket says 172.21.15.97:46442) 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cdc3b700 1 -- 172.21.15.97:0/3492487624 learned_addr learned my addr 172.21.15.97:0/3492487624 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.781+0000 7fa0cdc3b700 1 -- 172.21.15.97:0/3492487624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c4005040 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0cdc3b700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fa0c40089f0 tx=0x7fa0c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c4009320 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c8104a80 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0c4013380 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c4014e20 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.782+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c8104f10 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0c40095f0 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c8045ad0 con 0x7fa0c80f51a0 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0be7fc700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 0x7fa0b4034670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:13.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0c4040e20 con 0x7fa0c80f51a0 2024-02-19T22:23:13.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0cd43a700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 0x7fa0b4034670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:13.787 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.783+0000 7fa0cd43a700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 0x7fa0b4034670 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:13.790 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.788+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0c4014420 con 0x7fa0c80f51a0 2024-02-19T22:23:13.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:11.872744+0000 mgr.smithi045.bvlamo (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:13.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.995+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0c8105d60 con 0x7fa0c80f51a0 2024-02-19T22:23:13.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.996+0000 7fa0be7fc700 1 -- 172.21.15.97:0/3492487624 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa0c4010230 con 0x7fa0c80f51a0 2024-02-19T22:23:13.998 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:13.998 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 msgr2=0x7fa0b4034670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 0x7fa0b4034670 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fa0b80096a0 tx=0x7fa0b8006b40).stop 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 msgr2=0x7fa0c8104390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fa0c40089f0 tx=0x7fa0c400b040).stop 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 shutdown_connections 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa0b40321b0 0x7fa0b4034670 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 --2- 172.21.15.97:0/3492487624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa0c80f51a0 0x7fa0c8104390 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:14.000 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 >> 172.21.15.97:0/3492487624 conn(0x7fa0c80f09a0 msgr2=0x7fa0c80f13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:14.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 shutdown_connections 2024-02-19T22:23:14.001 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:13.997+0000 7fa0cec3d700 1 -- 172.21.15.97:0/3492487624 wait complete. 2024-02-19T22:23:14.001 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:14.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:13.996836+0000 mon.smithi045 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.97:0/3492487624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:15.651 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:15.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:13.873066+0000 mgr.smithi045.bvlamo (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:17.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.619+0000 7fb0f9926700 1 -- 172.21.15.97:0/967652711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 msgr2=0x7fb0f40f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.619+0000 7fb0f9926700 1 --2- 172.21.15.97:0/967652711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f40f4be0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fb0e4009fb0 tx=0x7fb0e4009b30).stop 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.619+0000 7fb0f9926700 1 -- 172.21.15.97:0/967652711 shutdown_connections 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.619+0000 7fb0f9926700 1 --2- 172.21.15.97:0/967652711 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f40f4be0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.619+0000 7fb0f9926700 1 -- 172.21.15.97:0/967652711 >> 172.21.15.97:0/967652711 conn(0x7fb0f40f02e0 msgr2=0x7fb0f40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 -- 172.21.15.97:0/967652711 shutdown_connections 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 -- 172.21.15.97:0/967652711 wait complete. 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 Processor -- start 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 -- start start 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:17.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.620+0000 7fb0f9926700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f4106cc0 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35194/0 (socket says 172.21.15.97:35194) 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f2ffd700 1 -- 172.21.15.97:0/1758705431 learned_addr learned my addr 172.21.15.97:0/1758705431 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f2ffd700 1 -- 172.21.15.97:0/1758705431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e4005040 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f2ffd700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb0e4005010 tx=0x7fb0e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e4009320 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0e4013380 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f4106ec0 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0e4014e50 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.621+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f4107350 con 0x7fb0f40f47c0 2024-02-19T22:23:17.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.622+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0e40095f0 con 0x7fb0f40f47c0 2024-02-19T22:23:17.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.622+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f4045ad0 con 0x7fb0f40f47c0 2024-02-19T22:23:17.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.622+0000 7fb0db7fe700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 0x7fb0dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:17.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.622+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0e4040f70 con 0x7fb0f40f47c0 2024-02-19T22:23:17.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.623+0000 7fb0f27fc700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 0x7fb0dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:17.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.623+0000 7fb0f27fc700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 0x7fb0dc0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fb0e80096a0 tx=0x7fb0e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:17.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.627+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0e4014420 con 0x7fb0f40f47c0 2024-02-19T22:23:17.832 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.830+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0f4108200 con 0x7fb0f40f47c0 2024-02-19T22:23:17.833 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.831+0000 7fb0db7fe700 1 -- 172.21.15.97:0/1758705431 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb0e401d980 con 0x7fb0f40f47c0 2024-02-19T22:23:17.833 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:17.834 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 msgr2=0x7fb0dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 0x7fb0dc0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fb0e80096a0 tx=0x7fb0e8006b40).stop 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 msgr2=0x7fb0f41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fb0e4005010 tx=0x7fb0e400b040).stop 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 shutdown_connections 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb0dc02ff10 0x7fb0dc0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:17.835 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.832+0000 7fb0f9926700 1 --2- 172.21.15.97:0/1758705431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb0f40f47c0 0x7fb0f41067d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.833+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 >> 172.21.15.97:0/1758705431 conn(0x7fb0f40f02e0 msgr2=0x7fb0f40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.833+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 shutdown_connections 2024-02-19T22:23:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:17.833+0000 7fb0f9926700 1 -- 172.21.15.97:0/1758705431 wait complete. 2024-02-19T22:23:17.836 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:18.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:15.873485+0000 mgr.smithi045.bvlamo (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:19.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:17.832172+0000 mon.smithi045 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.97:0/1758705431' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:19.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:19.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:20.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:17.873974+0000 mgr.smithi045.bvlamo (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 -- 172.21.15.97:0/1693381628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 msgr2=0x7effd80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 --2- 172.21.15.97:0/1693381628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd80f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7effc8009fb0 tx=0x7effc8009b30).stop 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 -- 172.21.15.97:0/1693381628 shutdown_connections 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 --2- 172.21.15.97:0/1693381628 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd80f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 -- 172.21.15.97:0/1693381628 >> 172.21.15.97:0/1693381628 conn(0x7effd80f1c70 msgr2=0x7effd80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 -- 172.21.15.97:0/1693381628 shutdown_connections 2024-02-19T22:23:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.380+0000 7effddee8700 1 -- 172.21.15.97:0/1693381628 wait complete. 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.381+0000 7effddee8700 1 Processor -- start 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.381+0000 7effddee8700 1 -- start start 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.381+0000 7effddee8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effddee8700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd8106cd0 con 0x7effd80f6510 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effd77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effd77fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35208/0 (socket says 172.21.15.97:35208) 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effd77fe700 1 -- 172.21.15.97:0/767239931 learned_addr learned my addr 172.21.15.97:0/767239931 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:21.386 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effd77fe700 1 -- 172.21.15.97:0/767239931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8005040 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effd77fe700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7effc8005010 tx=0x7effc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8009320 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd8106ed0 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc8013380 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effc8014e50 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.382+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd8107360 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.383+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7effc80263a0 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.383+0000 7effbffff700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 0x7effc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.383+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7effc8040f30 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.383+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd8045ad0 con 0x7effd80f6510 2024-02-19T22:23:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.384+0000 7effd6ffd700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 0x7effc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:21.388 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.384+0000 7effd6ffd700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 0x7effc00323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7effcc0096a0 tx=0x7effcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:21.390 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.388+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7effc8012920 con 0x7effd80f6510 2024-02-19T22:23:21.595 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.593+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7effd80fc190 con 0x7effd80f6510 2024-02-19T22:23:21.596 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.594+0000 7effbffff700 1 -- 172.21.15.97:0/767239931 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7effc802b020 con 0x7effd80f6510 2024-02-19T22:23:21.596 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:21.597 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 msgr2=0x7effc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 0x7effc00323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7effcc0096a0 tx=0x7effcc006b40).stop 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 msgr2=0x7effd81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7effc8005010 tx=0x7effc800b040).stop 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 shutdown_connections 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7effc002ff10 0x7effc00323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:21.598 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 --2- 172.21.15.97:0/767239931 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7effd80f6510 0x7effd81067e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:21.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 >> 172.21.15.97:0/767239931 conn(0x7effd80f1c70 msgr2=0x7effd80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:21.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 shutdown_connections 2024-02-19T22:23:21.599 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:21.596+0000 7effddee8700 1 -- 172.21.15.97:0/767239931 wait complete. 2024-02-19T22:23:21.599 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:22.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:19.874407+0000 mgr.smithi045.bvlamo (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:22.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:21.595157+0000 mon.smithi045 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.97:0/767239931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:23.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:23.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:24.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:21.874809+0000 mgr.smithi045.bvlamo (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:25.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 -- 172.21.15.97:0/697515611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 msgr2=0x7fc6240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:25.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 --2- 172.21.15.97:0/697515611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6240f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fc614009fb0 tx=0x7fc614009b30).stop 2024-02-19T22:23:25.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 -- 172.21.15.97:0/697515611 shutdown_connections 2024-02-19T22:23:25.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 --2- 172.21.15.97:0/697515611 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6240f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:25.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 -- 172.21.15.97:0/697515611 >> 172.21.15.97:0/697515611 conn(0x7fc6240f09a0 msgr2=0x7fc6240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:25.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 -- 172.21.15.97:0/697515611 shutdown_connections 2024-02-19T22:23:25.412 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 -- 172.21.15.97:0/697515611 wait complete. 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.409+0000 7fc623fff700 1 Processor -- start 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc623fff700 1 -- start start 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc623fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc623fff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6241059c0 con 0x7fc6240f51a0 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc622ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc622ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:35218/0 (socket says 172.21.15.97:35218) 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc622ffd700 1 -- 172.21.15.97:0/922703388 learned_addr learned my addr 172.21.15.97:0/922703388 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:25.413 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc622ffd700 1 -- 172.21.15.97:0/922703388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614005040 con 0x7fc6240f51a0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.410+0000 7fc622ffd700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fc6140089f0 tx=0x7fc61400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc614009320 con 0x7fc6240f51a0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc624105bc0 con 0x7fc6240f51a0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc614013380 con 0x7fc6240f51a0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc614014e20 con 0x7fc6240f51a0 2024-02-19T22:23:25.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc624106050 con 0x7fc6240f51a0 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.411+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc6140095f0 con 0x7fc6240f51a0 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.412+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc624045ad0 con 0x7fc6240f51a0 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.412+0000 7fc60b7fe700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 0x7fc60c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.412+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc614040f50 con 0x7fc6240f51a0 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.412+0000 7fc6227fc700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 0x7fc60c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:25.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.413+0000 7fc6227fc700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 0x7fc60c0323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc6180096a0 tx=0x7fc618006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:25.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.417+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc614014420 con 0x7fc6240f51a0 2024-02-19T22:23:25.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.629+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc624106f00 con 0x7fc6240f51a0 2024-02-19T22:23:25.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.630+0000 7fc60b7fe700 1 -- 172.21.15.97:0/922703388 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc61401d980 con 0x7fc6240f51a0 2024-02-19T22:23:25.632 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:25.632 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.631+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 msgr2=0x7fc60c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 0x7fc60c0323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fc6180096a0 tx=0x7fc618006b40).stop 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 msgr2=0x7fc6241054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fc6140089f0 tx=0x7fc61400b040).stop 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 shutdown_connections 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc60c02ff10 0x7fc60c0323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 --2- 172.21.15.97:0/922703388 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc6240f51a0 0x7fc6241054d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 >> 172.21.15.97:0/922703388 conn(0x7fc6240f09a0 msgr2=0x7fc6240f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 shutdown_connections 2024-02-19T22:23:25.634 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:25.632+0000 7fc623fff700 1 -- 172.21.15.97:0/922703388 wait complete. 2024-02-19T22:23:25.635 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:26.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:23.875227+0000 mgr.smithi045.bvlamo (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:26.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:25.631052+0000 mon.smithi045 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.97:0/922703388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:27.417 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:27.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:28.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:25.875641+0000 mgr.smithi045.bvlamo (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- 172.21.15.97:0/2245004457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 msgr2=0x7fa4e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/2245004457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00f55c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fa4dc009fb0 tx=0x7fa4dc009b30).stop 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- 172.21.15.97:0/2245004457 shutdown_connections 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/2245004457 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00f55c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- 172.21.15.97:0/2245004457 >> 172.21.15.97:0/2245004457 conn(0x7fa4e00f09a0 msgr2=0x7fa4e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- 172.21.15.97:0/2245004457 shutdown_connections 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- 172.21.15.97:0/2245004457 wait complete. 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 Processor -- start 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.494+0000 7fa4e60b1700 1 -- start start 2024-02-19T22:23:29.498 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e60b1700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e60b1700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00fb6f0 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e50af700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e50af700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41800/0 (socket says 172.21.15.97:41800) 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e50af700 1 -- 172.21.15.97:0/3804338942 learned_addr learned my addr 172.21.15.97:0/3804338942 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e50af700 1 -- 172.21.15.97:0/3804338942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc005040 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4e50af700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa4dc009f80 tx=0x7fa4dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.495+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4dc0095f0 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.496+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e00fb890 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.496+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4dc007e90 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.496+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4dc014d80 con 0x7fa4e00f51a0 2024-02-19T22:23:29.499 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.496+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e00fbd20 con 0x7fa4e00f51a0 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.497+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4dc025070 con 0x7fa4e00f51a0 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.497+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0059100 con 0x7fa4e00f51a0 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.497+0000 7fa4d5ffb700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 0x7fa4cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.497+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4dc040830 con 0x7fa4e00f51a0 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.497+0000 7fa4e48ae700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 0x7fa4cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:29.500 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.498+0000 7fa4e48ae700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 0x7fa4cc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:29.504 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.502+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4dc00f4e0 con 0x7fa4e00f51a0 2024-02-19T22:23:29.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.708+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e00fad30 con 0x7fa4e00f51a0 2024-02-19T22:23:29.710 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.708+0000 7fa4d5ffb700 1 -- 172.21.15.97:0/3804338942 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa4dc00f300 con 0x7fa4e00f51a0 2024-02-19T22:23:29.711 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:29.711 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 msgr2=0x7fa4cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 0x7fa4cc032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa4d00096a0 tx=0x7fa4d0006b40).stop 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 msgr2=0x7fa4e00fd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fa4dc009f80 tx=0x7fa4dc00b040).stop 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 shutdown_connections 2024-02-19T22:23:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa4cc02ff60 0x7fa4cc032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 --2- 172.21.15.97:0/3804338942 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa4e00f51a0 0x7fa4e00fd270 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 >> 172.21.15.97:0/3804338942 conn(0x7fa4e00f09a0 msgr2=0x7fa4e005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 shutdown_connections 2024-02-19T22:23:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:29.710+0000 7fa4e60b1700 1 -- 172.21.15.97:0/3804338942 wait complete. 2024-02-19T22:23:29.713 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:30.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:27.875964+0000 mgr.smithi045.bvlamo (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:29.709441+0000 mon.smithi045 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.97:0/3804338942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:31.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:31.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:32.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:29.876339+0000 mgr.smithi045.bvlamo (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:33.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3759039429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 msgr2=0x7f7dd00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:33.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3759039429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd00f55c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f7dc4009fb0 tx=0x7f7dc4009b30).stop 2024-02-19T22:23:33.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3759039429 shutdown_connections 2024-02-19T22:23:33.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3759039429 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd00f55c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3759039429 >> 172.21.15.97:0/3759039429 conn(0x7f7dd00f09a0 msgr2=0x7f7dd00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3759039429 shutdown_connections 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3759039429 wait complete. 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.520+0000 7f7dd5cfa700 1 Processor -- start 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd5cfa700 1 -- start start 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd5cfa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:33.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd5cfa700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0108d40 con 0x7f7dd00f51a0 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd4cf8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd4cf8700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:41824/0 (socket says 172.21.15.97:41824) 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.521+0000 7f7dd4cf8700 1 -- 172.21.15.97:0/3396718242 learned_addr learned my addr 172.21.15.97:0/3396718242 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dd4cf8700 1 -- 172.21.15.97:0/3396718242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc4005040 con 0x7f7dd00f51a0 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dd4cf8700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f7dc4009f80 tx=0x7f7dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:33.526 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc40095f0 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd0108f40 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dc4007e90 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc4014d80 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.522+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd01093d0 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.523+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7dc4025070 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.523+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd0045ad0 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.523+0000 7f7dcd7fa700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 0x7f7dc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.523+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7dc4040830 con 0x7f7dd00f51a0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.523+0000 7f7dcffff700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 0x7f7dc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.524+0000 7f7dcffff700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 0x7f7dc0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7dbc0096a0 tx=0x7f7dbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:33.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.528+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7dc400f4e0 con 0x7f7dd00f51a0 2024-02-19T22:23:33.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.738+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dd010a220 con 0x7f7dd00f51a0 2024-02-19T22:23:33.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.738+0000 7f7dcd7fa700 1 -- 172.21.15.97:0/3396718242 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7dc400f300 con 0x7f7dd00f51a0 2024-02-19T22:23:33.741 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:33.742 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 msgr2=0x7f7dc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 0x7f7dc0032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f7dbc0096a0 tx=0x7f7dbc006b40).stop 2024-02-19T22:23:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 msgr2=0x7f7dd0108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f7dc4009f80 tx=0x7f7dc400b040).stop 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 shutdown_connections 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7dc002ff60 0x7f7dc0032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 --2- 172.21.15.97:0/3396718242 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7dd00f51a0 0x7f7dd0108850 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 >> 172.21.15.97:0/3396718242 conn(0x7f7dd00f09a0 msgr2=0x7f7dd005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 shutdown_connections 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:33.740+0000 7f7dd5cfa700 1 -- 172.21.15.97:0/3396718242 wait complete. 2024-02-19T22:23:33.744 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:34.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:31.876798+0000 mgr.smithi045.bvlamo (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:35.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:33.739555+0000 mon.smithi045 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.97:0/3396718242' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:35.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:35.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:36.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:33.877269+0000 mgr.smithi045.bvlamo (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:37.120 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:35.877653+0000 mgr.smithi045.bvlamo (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/1970707447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 msgr2=0x7f7c740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/1970707447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c740f6930 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f7c64009fb0 tx=0x7f7c64009b30).stop 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/1970707447 shutdown_connections 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/1970707447 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c740f6930 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/1970707447 >> 172.21.15.97:0/1970707447 conn(0x7f7c740f1c70 msgr2=0x7f7c740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/1970707447 shutdown_connections 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.605+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/1970707447 wait complete. 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c7bd8d700 1 Processor -- start 2024-02-19T22:23:37.610 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c7bd8d700 1 -- start start 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c7bd8d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c7bd8d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74069e20 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c79b29700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c79b29700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50036/0 (socket says 172.21.15.97:50036) 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c79b29700 1 -- 172.21.15.97:0/2428649016 learned_addr learned my addr 172.21.15.97:0/2428649016 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.606+0000 7f7c79b29700 1 -- 172.21.15.97:0/2428649016 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c64005040 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c79b29700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7c640089f0 tx=0x7f7c6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c64009320 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c74069fc0 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c64013380 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c6401ed80 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.607+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7406a450 con 0x7f7c740f6510 2024-02-19T22:23:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c6401e420 con 0x7f7c740f6510 2024-02-19T22:23:37.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74059100 con 0x7f7c740f6510 2024-02-19T22:23:37.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c6a7fc700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 0x7f7c60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:37.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c64041090 con 0x7f7c740f6510 2024-02-19T22:23:37.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c79328700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 0x7f7c60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:37.612 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.608+0000 7f7c79328700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 0x7f7c60032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7c700096a0 tx=0x7f7c70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:37.614 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.612+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c640095f0 con 0x7f7c740f6510 2024-02-19T22:23:37.820 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.818+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c740fc0a0 con 0x7f7c740f6510 2024-02-19T22:23:37.821 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.819+0000 7f7c6a7fc700 1 -- 172.21.15.97:0/2428649016 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7c64004490 con 0x7f7c740f6510 2024-02-19T22:23:37.821 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:37.822 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 msgr2=0x7f7c60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 0x7f7c60032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f7c700096a0 tx=0x7f7c70006b40).stop 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 msgr2=0x7f7c7406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f7c640089f0 tx=0x7f7c6400b040).stop 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 shutdown_connections 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f7c6002ff60 0x7f7c60032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 --2- 172.21.15.97:0/2428649016 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f7c740f6510 0x7f7c7406b9a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 >> 172.21.15.97:0/2428649016 conn(0x7f7c740f1c70 msgr2=0x7f7c740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.820+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 shutdown_connections 2024-02-19T22:23:37.823 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:37.821+0000 7f7c7bd8d700 1 -- 172.21.15.97:0/2428649016 wait complete. 2024-02-19T22:23:37.824 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:38.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:37.819999+0000 mon.smithi045 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.97:0/2428649016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:39.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:39.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:39.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:37.878097+0000 mgr.smithi045.bvlamo (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.551+0000 7fe2a749c700 1 -- 172.21.15.97:0/3807808308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 msgr2=0x7fe2a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.551+0000 7fe2a749c700 1 --2- 172.21.15.97:0/3807808308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe290009fb0 tx=0x7fe290009b30).stop 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.551+0000 7fe2a749c700 1 -- 172.21.15.97:0/3807808308 shutdown_connections 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.551+0000 7fe2a749c700 1 --2- 172.21.15.97:0/3807808308 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.551+0000 7fe2a749c700 1 -- 172.21.15.97:0/3807808308 >> 172.21.15.97:0/3807808308 conn(0x7fe2a00f0b70 msgr2=0x7fe2a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 -- 172.21.15.97:0/3807808308 shutdown_connections 2024-02-19T22:23:41.554 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 -- 172.21.15.97:0/3807808308 wait complete. 2024-02-19T22:23:41.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 Processor -- start 2024-02-19T22:23:41.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 -- start start 2024-02-19T22:23:41.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a749c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2a00fb960 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a5238700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a5238700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50046/0 (socket says 172.21.15.97:50046) 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.552+0000 7fe2a5238700 1 -- 172.21.15.97:0/812654511 learned_addr learned my addr 172.21.15.97:0/812654511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe2a5238700 1 -- 172.21.15.97:0/812654511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe290005040 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe2a5238700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe290005010 tx=0x7fe29000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe290009320 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe290013380 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2a00fbb00 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe290014e50 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.553+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2a00fbf90 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.554+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe290014420 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.554+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2a0059100 con 0x7fe2a00f5410 2024-02-19T22:23:41.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.554+0000 7fe295ffb700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 0x7fe28c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:41.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.554+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe290040f40 con 0x7fe2a00f5410 2024-02-19T22:23:41.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.554+0000 7fe2a4a37700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 0x7fe28c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:41.558 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.555+0000 7fe2a4a37700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 0x7fe28c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fe29c0096a0 tx=0x7fe29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:41.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.559+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe2900095f0 con 0x7fe2a00f5410 2024-02-19T22:23:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.765+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2a00fafa0 con 0x7fe2a00f5410 2024-02-19T22:23:41.767 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.765+0000 7fe295ffb700 1 -- 172.21.15.97:0/812654511 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe29001db70 con 0x7fe2a00f5410 2024-02-19T22:23:41.768 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:41.768 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:41.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 msgr2=0x7fe28c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:41.770 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 0x7fe28c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fe29c0096a0 tx=0x7fe29c006b40).stop 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 msgr2=0x7fe2a00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fe290005010 tx=0x7fe29000b040).stop 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 shutdown_connections 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe28c02ff10 0x7fe28c0323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 --2- 172.21.15.97:0/812654511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe2a00f5410 0x7fe2a00fd4e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:41.771 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 >> 172.21.15.97:0/812654511 conn(0x7fe2a00f0b70 msgr2=0x7fe2a005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 shutdown_connections 2024-02-19T22:23:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:41.767+0000 7fe2a749c700 1 -- 172.21.15.97:0/812654511 wait complete. 2024-02-19T22:23:41.772 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:41.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:39.878451+0000 mgr.smithi045.bvlamo (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:42.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:41.766410+0000 mon.smithi045 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.97:0/812654511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:43.592 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:43.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:43.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:41.878770+0000 mgr.smithi045.bvlamo (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:45.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.527+0000 7f12669cd700 1 -- 172.21.15.97:0/1565029951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 msgr2=0x7f12600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.527+0000 7f12669cd700 1 --2- 172.21.15.97:0/1565029951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12600f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f1250009fb0 tx=0x7f1250009b30).stop 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.527+0000 7f12669cd700 1 -- 172.21.15.97:0/1565029951 shutdown_connections 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.527+0000 7f12669cd700 1 --2- 172.21.15.97:0/1565029951 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12600f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 -- 172.21.15.97:0/1565029951 >> 172.21.15.97:0/1565029951 conn(0x7f12600f09a0 msgr2=0x7f12600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 -- 172.21.15.97:0/1565029951 shutdown_connections 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 -- 172.21.15.97:0/1565029951 wait complete. 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 Processor -- start 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 -- start start 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:45.532 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.528+0000 7f12669cd700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1260106ad0 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12659cb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12659cb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:50068/0 (socket says 172.21.15.97:50068) 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12659cb700 1 -- 172.21.15.97:0/2321418684 learned_addr learned my addr 172.21.15.97:0/2321418684 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12659cb700 1 -- 172.21.15.97:0/2321418684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1250005040 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12659cb700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f12500089f0 tx=0x7f125000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1250009320 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1260106cd0 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1250013380 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1250014e20 con 0x7f12600f51a0 2024-02-19T22:23:45.533 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.529+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1260107160 con 0x7f12600f51a0 2024-02-19T22:23:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.530+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12500095f0 con 0x7f12600f51a0 2024-02-19T22:23:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.530+0000 7f12567fc700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.530+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1250040f50 con 0x7f12600f51a0 2024-02-19T22:23:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.530+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1260045ad0 con 0x7f12600f51a0 2024-02-19T22:23:45.534 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.531+0000 7f12651ca700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:45.535 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.531+0000 7f12651ca700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 0x7f124c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f125c0096a0 tx=0x7f125c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:45.537 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.535+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1250014420 con 0x7f12600f51a0 2024-02-19T22:23:45.741 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.739+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1260108010 con 0x7f12600f51a0 2024-02-19T22:23:45.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.740+0000 7f12567fc700 1 -- 172.21.15.97:0/2321418684 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f125001d980 con 0x7f12600f51a0 2024-02-19T22:23:45.742 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:45.743 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:45.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 msgr2=0x7f124c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:45.744 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 0x7f124c032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f125c0096a0 tx=0x7f125c006b40).stop 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 msgr2=0x7f12601065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f12500089f0 tx=0x7f125000b040).stop 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 shutdown_connections 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f124c02ff60 0x7f124c032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 --2- 172.21.15.97:0/2321418684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f12600f51a0 0x7f12601065e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 >> 172.21.15.97:0/2321418684 conn(0x7f12600f09a0 msgr2=0x7f12600f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 shutdown_connections 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:45.742+0000 7f12669cd700 1 -- 172.21.15.97:0/2321418684 wait complete. 2024-02-19T22:23:45.745 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:45.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:45 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:43.879082+0000 mgr.smithi045.bvlamo (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:46.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:45.741303+0000 mon.smithi045 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.97:0/2321418684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:47.443 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:47.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:47.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:45.879477+0000 mgr.smithi045.bvlamo (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:49.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 -- 172.21.15.97:0/984077173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 msgr2=0x7f41d40f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 --2- 172.21.15.97:0/984077173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d40f9190 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f41c8009fb0 tx=0x7f41c8009b30).stop 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 -- 172.21.15.97:0/984077173 shutdown_connections 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 --2- 172.21.15.97:0/984077173 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d40f9190 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 -- 172.21.15.97:0/984077173 >> 172.21.15.97:0/984077173 conn(0x7f41d40f0b70 msgr2=0x7f41d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 -- 172.21.15.97:0/984077173 shutdown_connections 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.451+0000 7f41d8aff700 1 -- 172.21.15.97:0/984077173 wait complete. 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.452+0000 7f41d8aff700 1 Processor -- start 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.452+0000 7f41d8aff700 1 -- start start 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.452+0000 7f41d8aff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.452+0000 7f41d8aff700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d4106cc0 con 0x7f41d40601e0 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:49.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44294/0 (socket says 172.21.15.97:44294) 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d259c700 1 -- 172.21.15.97:0/1746740087 learned_addr learned my addr 172.21.15.97:0/1746740087 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d259c700 1 -- 172.21.15.97:0/1746740087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8005040 con 0x7f41d40601e0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d259c700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f41c8007a10 tx=0x7f41c8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c8018070 con 0x7f41d40601e0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d4106ec0 con 0x7f41d40601e0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f41c8013410 con 0x7f41d40601e0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41c801c3c0 con 0x7f41d40601e0 2024-02-19T22:23:49.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.453+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d4107350 con 0x7f41d40601e0 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.454+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f41c801c680 con 0x7f41d40601e0 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.454+0000 7f41c2ffd700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 0x7f41bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.454+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f41c8041540 con 0x7f41d40601e0 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.454+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d4045ad0 con 0x7f41d40601e0 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.454+0000 7f41d1d9b700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 0x7f41bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:49.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.455+0000 7f41d1d9b700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 0x7f41bc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f41c40096a0 tx=0x7f41c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.459+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41c801ca80 con 0x7f41d40601e0 2024-02-19T22:23:49.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.666+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41d4108200 con 0x7f41d40601e0 2024-02-19T22:23:49.668 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.666+0000 7f41c2ffd700 1 -- 172.21.15.97:0/1746740087 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f41c801b9c0 con 0x7f41d40601e0 2024-02-19T22:23:49.669 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:49.669 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:49.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 msgr2=0x7f41bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:49.670 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 0x7f41bc032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f41c40096a0 tx=0x7f41c4006b40).stop 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 msgr2=0x7f41d41067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f41c8007a10 tx=0x7f41c8009b30).stop 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 shutdown_connections 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f41bc02ff60 0x7f41bc032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 --2- 172.21.15.97:0/1746740087 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f41d40601e0 0x7f41d41067d0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 >> 172.21.15.97:0/1746740087 conn(0x7f41d40f0b70 msgr2=0x7f41d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 shutdown_connections 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:49.668+0000 7f41d8aff700 1 -- 172.21.15.97:0/1746740087 wait complete. 2024-02-19T22:23:49.671 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:49.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:47.879840+0000 mgr.smithi045.bvlamo (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:50.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:49.667406+0000 mon.smithi045 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.97:0/1746740087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:51.424 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:51.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:51.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:49.880200+0000 mgr.smithi045.bvlamo (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:53.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- 172.21.15.97:0/2267997391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 msgr2=0x7f47d00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:53.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 --2- 172.21.15.97:0/2267997391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d00f66c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f47c0009fb0 tx=0x7f47c0009b30).stop 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- 172.21.15.97:0/2267997391 shutdown_connections 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 --2- 172.21.15.97:0/2267997391 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d00f66c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- 172.21.15.97:0/2267997391 >> 172.21.15.97:0/2267997391 conn(0x7f47d00f1aa0 msgr2=0x7f47d00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- 172.21.15.97:0/2267997391 shutdown_connections 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- 172.21.15.97:0/2267997391 wait complete. 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 Processor -- start 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- start start 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.487+0000 7f47cf59e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d0106ac0 con 0x7f47d00f62a0 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47ce59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47ce59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44310/0 (socket says 172.21.15.97:44310) 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47ce59c700 1 -- 172.21.15.97:0/160658624 learned_addr learned my addr 172.21.15.97:0/160658624 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47ce59c700 1 -- 172.21.15.97:0/160658624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c0005040 con 0x7f47d00f62a0 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47ce59c700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f47c00089f0 tx=0x7f47c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:53.491 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47c0009320 con 0x7f47d00f62a0 2024-02-19T22:23:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d0106cc0 con 0x7f47d00f62a0 2024-02-19T22:23:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f47c0013380 con 0x7f47d00f62a0 2024-02-19T22:23:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47c0014e20 con 0x7f47d00f62a0 2024-02-19T22:23:53.492 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.488+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d0107150 con 0x7f47d00f62a0 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.489+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f47c00095f0 con 0x7f47d00f62a0 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.489+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d0045ad0 con 0x7f47d00f62a0 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.489+0000 7f47beffd700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 0x7f47b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.489+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47c0040e90 con 0x7f47d00f62a0 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.489+0000 7f47cdd9b700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 0x7f47b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:53.493 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.490+0000 7f47cdd9b700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 0x7f47b80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f47c40096a0 tx=0x7f47c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:53.496 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.494+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f47c0014420 con 0x7f47d00f62a0 2024-02-19T22:23:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.700+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47d0108000 con 0x7f47d00f62a0 2024-02-19T22:23:53.702 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.700+0000 7f47beffd700 1 -- 172.21.15.97:0/160658624 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f47c001d980 con 0x7f47d00f62a0 2024-02-19T22:23:53.703 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:53.703 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 msgr2=0x7f47b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 0x7f47b80323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f47c40096a0 tx=0x7f47c4006b40).stop 2024-02-19T22:23:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 msgr2=0x7f47d01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:53.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f47c00089f0 tx=0x7f47c000b040).stop 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 shutdown_connections 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f47b802ff10 0x7f47b80323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 --2- 172.21.15.97:0/160658624 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f47d00f62a0 0x7f47d01065d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 >> 172.21.15.97:0/160658624 conn(0x7f47d00f1aa0 msgr2=0x7f47d00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 shutdown_connections 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:53.702+0000 7f47cf59e700 1 -- 172.21.15.97:0/160658624 wait complete. 2024-02-19T22:23:53.705 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:53.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:51.880620+0000 mgr.smithi045.bvlamo (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:54.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:53.701482+0000 mon.smithi045 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.97:0/160658624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:55.333 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:55.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:55.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:53.881040+0000 mgr.smithi045.bvlamo (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:57.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 -- 172.21.15.97:0/2900461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 msgr2=0x7f89a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:57.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/2900461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a00f5830 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f8994009fb0 tx=0x7f8994009b30).stop 2024-02-19T22:23:57.236 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 -- 172.21.15.97:0/2900461 shutdown_connections 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/2900461 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a00f5830 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 -- 172.21.15.97:0/2900461 >> 172.21.15.97:0/2900461 conn(0x7f89a00f0b70 msgr2=0x7f89a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 -- 172.21.15.97:0/2900461 shutdown_connections 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.232+0000 7f89a6fa4700 1 -- 172.21.15.97:0/2900461 wait complete. 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a6fa4700 1 Processor -- start 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a6fa4700 1 -- start start 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a6fa4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a6fa4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a0060240 con 0x7f89a00f5410 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a4d40700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.233+0000 7f89a4d40700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55752/0 (socket says 172.21.15.97:55752) 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f89a4d40700 1 -- 172.21.15.97:0/504201664 learned_addr learned my addr 172.21.15.97:0/504201664 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f89a4d40700 1 -- 172.21.15.97:0/504201664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994005040 con 0x7f89a00f5410 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f89a4d40700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8994005010 tx=0x7f899400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:57.237 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8994009320 con 0x7f89a00f5410 2024-02-19T22:23:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a0060440 con 0x7f89a00f5410 2024-02-19T22:23:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8994013380 con 0x7f89a00f5410 2024-02-19T22:23:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8994014e50 con 0x7f89a00f5410 2024-02-19T22:23:57.238 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.234+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a0109b90 con 0x7f89a00f5410 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.235+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89940095f0 con 0x7f89a00f5410 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.235+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a0045ad0 con 0x7f89a00f5410 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.235+0000 7f899d7fa700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 0x7f89900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.235+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8994040f70 con 0x7f89a00f5410 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.236+0000 7f899ffff700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 0x7f89900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:57.239 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.236+0000 7f899ffff700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 0x7f89900323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f898c0096a0 tx=0x7f898c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:57.242 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.240+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8994014420 con 0x7f89a00f5410 2024-02-19T22:23:57.448 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.446+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89a010a3b0 con 0x7f89a00f5410 2024-02-19T22:23:57.449 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.447+0000 7f899d7fa700 1 -- 172.21.15.97:0/504201664 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8994010230 con 0x7f89a00f5410 2024-02-19T22:23:57.449 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:23:57.449 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.448+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 msgr2=0x7f89900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.448+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 0x7f89900323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f898c0096a0 tx=0x7f898c006b40).stop 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.448+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 msgr2=0x7f89a01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.448+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f8994005010 tx=0x7f899400b040).stop 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 shutdown_connections 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f899002ff10 0x7f89900323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 --2- 172.21.15.97:0/504201664 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89a00f5410 0x7f89a01094e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 >> 172.21.15.97:0/504201664 conn(0x7f89a00f0b70 msgr2=0x7f89a00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 shutdown_connections 2024-02-19T22:23:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:23:57.449+0000 7f89a6fa4700 1 -- 172.21.15.97:0/504201664 wait complete. 2024-02-19T22:23:57.452 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:23:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:55.881356+0000 mgr.smithi045.bvlamo (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:57.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:23:57.447996+0000 mon.smithi045 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.97:0/504201664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:59.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:59.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:23:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:57.881712+0000 mgr.smithi045.bvlamo (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.188+0000 7febcf499700 1 -- 172.21.15.97:0/50038371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 msgr2=0x7febc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.188+0000 7febcf499700 1 --2- 172.21.15.97:0/50038371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc80f55c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7febb8009fb0 tx=0x7febb8009b30).stop 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- 172.21.15.97:0/50038371 shutdown_connections 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 --2- 172.21.15.97:0/50038371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc80f55c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- 172.21.15.97:0/50038371 >> 172.21.15.97:0/50038371 conn(0x7febc80f09a0 msgr2=0x7febc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- 172.21.15.97:0/50038371 shutdown_connections 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- 172.21.15.97:0/50038371 wait complete. 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 Processor -- start 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- start start 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febcf499700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc81059c0 con 0x7febc80f51a0 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febce497700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febce497700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55782/0 (socket says 172.21.15.97:55782) 2024-02-19T22:24:01.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.189+0000 7febce497700 1 -- 172.21.15.97:0/1853482608 learned_addr learned my addr 172.21.15.97:0/1853482608 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febce497700 1 -- 172.21.15.97:0/1853482608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8005040 con 0x7febc80f51a0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febce497700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7febb80089f0 tx=0x7febb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febb8009320 con 0x7febc80f51a0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc8105bc0 con 0x7febc80f51a0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7febb8013380 con 0x7febc80f51a0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7febb8014e20 con 0x7febc80f51a0 2024-02-19T22:24:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.190+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc8106050 con 0x7febc80f51a0 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.191+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7febb80263a0 con 0x7febc80f51a0 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.191+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc8045ad0 con 0x7febc80f51a0 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.191+0000 7febbeffd700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 0x7febb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.191+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7febb80411a0 con 0x7febc80f51a0 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.191+0000 7febcdc96700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 0x7febb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.192+0000 7febcdc96700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 0x7febb40323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7febc40096a0 tx=0x7febc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:01.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.196+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7febb80126f0 con 0x7febc80f51a0 2024-02-19T22:24:01.405 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.403+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7febc8106f00 con 0x7febc80f51a0 2024-02-19T22:24:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.403+0000 7febbeffd700 1 -- 172.21.15.97:0/1853482608 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7febb80126f0 con 0x7febc80f51a0 2024-02-19T22:24:01.406 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:01.406 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 msgr2=0x7febb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 0x7febb40323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7febc40096a0 tx=0x7febc4006b40).stop 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 msgr2=0x7febc81054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7febb80089f0 tx=0x7febb800b040).stop 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 shutdown_connections 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7febb402ff10 0x7febb40323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 --2- 172.21.15.97:0/1853482608 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7febc80f51a0 0x7febc81054d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 >> 172.21.15.97:0/1853482608 conn(0x7febc80f09a0 msgr2=0x7febc80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 shutdown_connections 2024-02-19T22:24:01.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:01.405+0000 7febcf499700 1 -- 172.21.15.97:0/1853482608 wait complete. 2024-02-19T22:24:01.409 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:02.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:23:59.882011+0000 mgr.smithi045.bvlamo (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:02.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:01.404574+0000 mon.smithi045 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.97:0/1853482608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:03.049 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:03.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:01.882332+0000 mgr.smithi045.bvlamo (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:02.755223+0000 mon.smithi045 (mon.0) 288 : audit [DBG] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:24:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:02.991044+0000 mon.smithi045 (mon.0) 289 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:02.993056+0000 mon.smithi045 (mon.0) 290 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:03.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:03 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:03.233101+0000 mon.smithi045 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:24:02.754588+0000 mgr.smithi045.bvlamo (mgr.14162) 308 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi045 ... 2024-02-19T22:24:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:22:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:22:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:23:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:24:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('uid/gid not found') 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: uid/gid not found 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: yield (conn, connr) 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: code, '\n'.join(err))) 2024-02-19T22:24:04.893 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi045 ... 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi045 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:22:10Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:22:47Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr time="2024-02-19T22:23:25Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: Traceback (most recent call last): 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: main() 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: r = ctx.func(ctx) 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:24:04.894 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: return func(ctx) 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: File "/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: raise RuntimeError('uid/gid not found') 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: RuntimeError: uid/gid not found 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:03.939741+0000 mon.smithi045 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:24:04.895 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:03.939933+0000 mon.smithi045 (mon.0) 293 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:24:06.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:03.882641+0000 mgr.smithi045.bvlamo (mgr.14162) 309 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:06.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:05.090049+0000 mon.smithi045 (mon.0) 294 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:06.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.419+0000 7f3f93fad700 1 -- 172.21.15.97:0/4152751371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 msgr2=0x7f3f8c0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.419+0000 7f3f93fad700 1 --2- 172.21.15.97:0/4152751371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c0f9150 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f3f88009fb0 tx=0x7f3f88009b30).stop 2024-02-19T22:24:06.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.419+0000 7f3f93fad700 1 -- 172.21.15.97:0/4152751371 shutdown_connections 2024-02-19T22:24:06.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.419+0000 7f3f93fad700 1 --2- 172.21.15.97:0/4152751371 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c0f9150 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.419+0000 7f3f93fad700 1 -- 172.21.15.97:0/4152751371 >> 172.21.15.97:0/4152751371 conn(0x7f3f8c0f0b70 msgr2=0x7f3f8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 -- 172.21.15.97:0/4152751371 shutdown_connections 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 -- 172.21.15.97:0/4152751371 wait complete. 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 Processor -- start 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 -- start start 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f93fad700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c106c90 con 0x7f3f8c060220 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f91d49700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f91d49700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44726/0 (socket says 172.21.15.97:44726) 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.420+0000 7f3f91d49700 1 -- 172.21.15.97:0/3512163101 learned_addr learned my addr 172.21.15.97:0/3512163101 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f91d49700 1 -- 172.21.15.97:0/3512163101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f88005040 con 0x7f3f8c060220 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f91d49700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3f880089f0 tx=0x7f3f8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f88009320 con 0x7f3f8c060220 2024-02-19T22:24:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c106e90 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c107320 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f88013380 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.421+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f88014db0 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.422+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f74004fa0 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.422+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f880095f0 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.422+0000 7f3f827fc700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.422+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f88040e40 con 0x7f3f8c060220 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.423+0000 7f3f91548700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.423+0000 7f3f91548700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 0x7f3f78032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f3f7c0096a0 tx=0x7f3f7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:06.457 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.427+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f88004490 con 0x7f3f8c060220 2024-02-19T22:24:06.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.642+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f74005670 con 0x7f3f8c060220 2024-02-19T22:24:06.645 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.643+0000 7f3f827fc700 1 -- 172.21.15.97:0/3512163101 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3f88012330 con 0x7f3f8c060220 2024-02-19T22:24:06.646 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:06.646 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 msgr2=0x7f3f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 0x7f3f78032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f3f7c0096a0 tx=0x7f3f7c006b40).stop 2024-02-19T22:24:06.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 msgr2=0x7f3f8c1067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3f880089f0 tx=0x7f3f8800b040).stop 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 shutdown_connections 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f3f7802ff60 0x7f3f78032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 --2- 172.21.15.97:0/3512163101 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f3f8c060220 0x7f3f8c1067a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 >> 172.21.15.97:0/3512163101 conn(0x7f3f8c0f0b70 msgr2=0x7f3f8c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 shutdown_connections 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:06.645+0000 7f3f93fad700 1 -- 172.21.15.97:0/3512163101 wait complete. 2024-02-19T22:24:06.648 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:07.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:06.644258+0000 mon.smithi045 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.97:0/3512163101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:08.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:07 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:05.882971+0000 mgr.smithi045.bvlamo (mgr.14162) 310 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:08.399 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:08.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:09.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:07.883324+0000 mgr.smithi045.bvlamo (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:09.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:08.193730+0000 mon.smithi045 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:09.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:08.794645+0000 mon.smithi045 (mon.0) 297 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:09.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:08.797433+0000 mon.smithi045 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' 2024-02-19T22:24:10.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.534+0000 7f57f7fcf700 1 -- 172.21.15.97:0/4260810501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 msgr2=0x7f57f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.534+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/4260810501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f57ec009fb0 tx=0x7f57ec009b30).stop 2024-02-19T22:24:10.538 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- 172.21.15.97:0/4260810501 shutdown_connections 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/4260810501 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- 172.21.15.97:0/4260810501 >> 172.21.15.97:0/4260810501 conn(0x7f57f00f0b70 msgr2=0x7f57f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- 172.21.15.97:0/4260810501 shutdown_connections 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- 172.21.15.97:0/4260810501 wait complete. 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 Processor -- start 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- start start 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.535+0000 7f57f7fcf700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f00fb960 con 0x7f57f00f5410 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f5d6b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f5d6b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44742/0 (socket says 172.21.15.97:44742) 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f5d6b700 1 -- 172.21.15.97:0/2476628422 learned_addr learned my addr 172.21.15.97:0/2476628422 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:10.539 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f5d6b700 1 -- 172.21.15.97:0/2476628422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec005040 con 0x7f57f00f5410 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f5d6b700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f57ec005010 tx=0x7f57ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ec009320 con 0x7f57f00f5410 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f00fbb00 con 0x7f57f00f5410 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57ec013380 con 0x7f57f00f5410 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57ec014e50 con 0x7f57f00f5410 2024-02-19T22:24:10.540 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.536+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f00fbf90 con 0x7f57f00f5410 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.537+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57ec0095f0 con 0x7f57f00f5410 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.537+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f0059100 con 0x7f57f00f5410 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.537+0000 7f57e67fc700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 0x7f57dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.537+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57ec040f70 con 0x7f57f00f5410 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.537+0000 7f57f556a700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 0x7f57dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:10.541 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.538+0000 7f57f556a700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 0x7f57dc0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f57e00096a0 tx=0x7f57e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:10.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.542+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57ec014420 con 0x7f57f00f5410 2024-02-19T22:24:10.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:08.795380+0000 mgr.smithi045.bvlamo (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:10.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cephadm 2024-02-19T22:24:08.799579+0000 mgr.smithi045.bvlamo (mgr.14162) 313 : cephadm [INF] Deploying daemon alertmanager.smithi097 on smithi097 2024-02-19T22:24:10.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:09.793284+0000 mon.smithi045 (mon.0) 299 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:24:10.751 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.749+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57f0045ad0 con 0x7f57f00f5410 2024-02-19T22:24:10.752 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.750+0000 7f57e67fc700 1 -- 172.21.15.97:0/2476628422 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57ec010230 con 0x7f57f00f5410 2024-02-19T22:24:10.752 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:10.752 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 msgr2=0x7f57dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 0x7f57dc0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f57e00096a0 tx=0x7f57e0006b40).stop 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 msgr2=0x7f57f00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f57ec005010 tx=0x7f57ec00b040).stop 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 shutdown_connections 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f57dc02ff10 0x7f57dc0323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 --2- 172.21.15.97:0/2476628422 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f57f00f5410 0x7f57f00fd4e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.754 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.751+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 >> 172.21.15.97:0/2476628422 conn(0x7f57f00f0b70 msgr2=0x7f57f005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:10.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.752+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 shutdown_connections 2024-02-19T22:24:10.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:10.752+0000 7f57f7fcf700 1 -- 172.21.15.97:0/2476628422 wait complete. 2024-02-19T22:24:10.755 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:11.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:11 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:10.750558+0000 mon.smithi045 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.97:0/2476628422' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:12.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:12.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:12.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:12 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:10.795731+0000 mgr.smithi045.bvlamo (mgr.14162) 314 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:14.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- 172.21.15.97:0/411240989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 msgr2=0x7f35580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 --2- 172.21.15.97:0/411240989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35580f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f3548009fb0 tx=0x7f3548009b30).stop 2024-02-19T22:24:14.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- 172.21.15.97:0/411240989 shutdown_connections 2024-02-19T22:24:14.622 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 --2- 172.21.15.97:0/411240989 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35580f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- 172.21.15.97:0/411240989 >> 172.21.15.97:0/411240989 conn(0x7f35580f0b70 msgr2=0x7f35580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- 172.21.15.97:0/411240989 shutdown_connections 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- 172.21.15.97:0/411240989 wait complete. 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 Processor -- start 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.618+0000 7f355dac4700 1 -- start start 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f355dac4700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f355dac4700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3558106ca0 con 0x7f35580f5410 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f35577fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f35577fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:44752/0 (socket says 172.21.15.97:44752) 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f35577fe700 1 -- 172.21.15.97:0/3182438636 learned_addr learned my addr 172.21.15.97:0/3182438636 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f35577fe700 1 -- 172.21.15.97:0/3182438636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3548005040 con 0x7f35580f5410 2024-02-19T22:24:14.623 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.619+0000 7f35577fe700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f3548005010 tx=0x7f354800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.620+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3548009320 con 0x7f35580f5410 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.620+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3558106ea0 con 0x7f35580f5410 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.620+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3558107330 con 0x7f35580f5410 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.620+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3558045ad0 con 0x7f35580f5410 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.622+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3548013380 con 0x7f35580f5410 2024-02-19T22:24:14.624 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.622+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3548027cb0 con 0x7f35580f5410 2024-02-19T22:24:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.623+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3548027430 con 0x7f35580f5410 2024-02-19T22:24:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.623+0000 7f3554ff9700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 0x7f3544032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.623+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3548041280 con 0x7f35580f5410 2024-02-19T22:24:14.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.623+0000 7f354ffff700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 0x7f3544032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:14.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.623+0000 7f354ffff700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 0x7f3544032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f35400096a0 tx=0x7f3540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:14.629 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.627+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3548027e10 con 0x7f35580f5410 2024-02-19T22:24:14.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.835+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35580fb090 con 0x7f35580f5410 2024-02-19T22:24:14.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.836+0000 7f3554ff9700 1 -- 172.21.15.97:0/3182438636 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f354800f990 con 0x7f35580f5410 2024-02-19T22:24:14.839 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:14.839 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:14.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 msgr2=0x7f3544032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 0x7f3544032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f35400096a0 tx=0x7f3540006b40).stop 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 msgr2=0x7f35581067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f3548005010 tx=0x7f354800b040).stop 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 shutdown_connections 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f354402ffb0 0x7f3544032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 --2- 172.21.15.97:0/3182438636 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f35580f5410 0x7f35581067b0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 >> 172.21.15.97:0/3182438636 conn(0x7f35580f0b70 msgr2=0x7f355805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 shutdown_connections 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:14.838+0000 7f355dac4700 1 -- 172.21.15.97:0/3182438636 wait complete. 2024-02-19T22:24:14.841 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:14.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:12.796108+0000 mgr.smithi045.bvlamo (mgr.14162) 315 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:15.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:15 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:14.837218+0000 mon.smithi045 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.97:0/3182438636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:16.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:16.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:16.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:14.796369+0000 mgr.smithi045.bvlamo (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:18.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 -- 172.21.15.97:0/2246094210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 msgr2=0x7f07240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:18.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 --2- 172.21.15.97:0/2246094210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07240f55c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f0720009fb0 tx=0x7f0720009b30).stop 2024-02-19T22:24:18.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 -- 172.21.15.97:0/2246094210 shutdown_connections 2024-02-19T22:24:18.677 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 --2- 172.21.15.97:0/2246094210 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07240f55c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 -- 172.21.15.97:0/2246094210 >> 172.21.15.97:0/2246094210 conn(0x7f07240f09a0 msgr2=0x7f07240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 -- 172.21.15.97:0/2246094210 shutdown_connections 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.673+0000 7f072a6de700 1 -- 172.21.15.97:0/2246094210 wait complete. 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f072a6de700 1 Processor -- start 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f072a6de700 1 -- start start 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f072a6de700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f072a6de700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0724106ac0 con 0x7f07240f51a0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f07296dc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f07296dc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:43108/0 (socket says 172.21.15.97:43108) 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.674+0000 7f07296dc700 1 -- 172.21.15.97:0/1441810650 learned_addr learned my addr 172.21.15.97:0/1441810650 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f07296dc700 1 -- 172.21.15.97:0/1441810650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720005040 con 0x7f07240f51a0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f07296dc700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f07200089f0 tx=0x7f072000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0720009320 con 0x7f07240f51a0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0724106cc0 con 0x7f07240f51a0 2024-02-19T22:24:18.678 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0720013380 con 0x7f07240f51a0 2024-02-19T22:24:18.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0720014e20 con 0x7f07240f51a0 2024-02-19T22:24:18.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.675+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0724107150 con 0x7f07240f51a0 2024-02-19T22:24:18.679 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.676+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f07200095f0 con 0x7f07240f51a0 2024-02-19T22:24:18.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.676+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0724045ad0 con 0x7f07240f51a0 2024-02-19T22:24:18.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.676+0000 7f0719ffb700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 0x7f07100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:18.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.676+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0720040e90 con 0x7f07240f51a0 2024-02-19T22:24:18.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.676+0000 7f0728edb700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 0x7f07100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:18.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.677+0000 7f0728edb700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 0x7f07100323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f07140096a0 tx=0x7f0714006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:18.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.681+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0720014420 con 0x7f07240f51a0 2024-02-19T22:24:18.890 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.888+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07240fad30 con 0x7f07240f51a0 2024-02-19T22:24:18.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:16.796740+0000 mgr.smithi045.bvlamo (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:18.891 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.889+0000 7f0719ffb700 1 -- 172.21.15.97:0/1441810650 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f072001d980 con 0x7f07240f51a0 2024-02-19T22:24:18.891 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:18.891 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 msgr2=0x7f07100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 0x7f07100323d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f07140096a0 tx=0x7f0714006b40).stop 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 msgr2=0x7f07241065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f07200089f0 tx=0x7f072000b040).stop 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 shutdown_connections 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f071002ff10 0x7f07100323d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 --2- 172.21.15.97:0/1441810650 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f07240f51a0 0x7f07241065d0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 >> 172.21.15.97:0/1441810650 conn(0x7f07240f09a0 msgr2=0x7f072405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:18.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 shutdown_connections 2024-02-19T22:24:18.894 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:18.890+0000 7f072a6de700 1 -- 172.21.15.97:0/1441810650 wait complete. 2024-02-19T22:24:18.894 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:19.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:19 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:18.889696+0000 mon.smithi045 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.97:0/1441810650' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:20.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:20.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:20.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:18.797141+0000 mgr.smithi045.bvlamo (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- 172.21.15.97:0/3276973934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 msgr2=0x7f28a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 --2- 172.21.15.97:0/3276973934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f2890009fb0 tx=0x7f2890009b30).stop 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- 172.21.15.97:0/3276973934 shutdown_connections 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 --2- 172.21.15.97:0/3276973934 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- 172.21.15.97:0/3276973934 >> 172.21.15.97:0/3276973934 conn(0x7f28a00f0b70 msgr2=0x7f28a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- 172.21.15.97:0/3276973934 shutdown_connections 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- 172.21.15.97:0/3276973934 wait complete. 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 Processor -- start 2024-02-19T22:24:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.680+0000 7f28a49ec700 1 -- start start 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f28a49ec700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f28a49ec700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00fb960 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f289e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f289e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:43118/0 (socket says 172.21.15.97:43118) 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f289e59c700 1 -- 172.21.15.97:0/1387963872 learned_addr learned my addr 172.21.15.97:0/1387963872 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f289e59c700 1 -- 172.21.15.97:0/1387963872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2890005040 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.681+0000 7f289e59c700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2890005010 tx=0x7f289000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2890009320 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a00fbb00 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2890013380 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2890014e50 con 0x7f28a00f5410 2024-02-19T22:24:22.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a00fbf90 con 0x7f28a00f5410 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28900095f0 con 0x7f28a00f5410 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.682+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a0059100 con 0x7f28a00f5410 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.683+0000 7f288effd700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 0x7f28880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.683+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2890040e90 con 0x7f28a00f5410 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.683+0000 7f289dd9b700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 0x7f28880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:22.686 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.683+0000 7f289dd9b700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 0x7f28880323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f28940096a0 tx=0x7f2894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:22.689 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.687+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2890014420 con 0x7f28a00f5410 2024-02-19T22:24:22.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:20.797401+0000 mgr.smithi045.bvlamo (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:22.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.890+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28a00fafa0 con 0x7f28a00f5410 2024-02-19T22:24:22.893 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.891+0000 7f288effd700 1 -- 172.21.15.97:0/1387963872 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2890010230 con 0x7f28a00f5410 2024-02-19T22:24:22.893 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:22.894 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 msgr2=0x7f28880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 0x7f28880323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f28940096a0 tx=0x7f2894006b40).stop 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 msgr2=0x7f28a00fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2890005010 tx=0x7f289000b040).stop 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 shutdown_connections 2024-02-19T22:24:22.895 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f288802ff10 0x7f28880323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:22.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 --2- 172.21.15.97:0/1387963872 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f28a00f5410 0x7f28a00fd4e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:22.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 >> 172.21.15.97:0/1387963872 conn(0x7f28a00f0b70 msgr2=0x7f28a005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:22.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 shutdown_connections 2024-02-19T22:24:22.896 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:22.893+0000 7f28a49ec700 1 -- 172.21.15.97:0/1387963872 wait complete. 2024-02-19T22:24:22.896 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:23.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:23 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:22.892246+0000 mon.smithi045 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.97:0/1387963872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:24.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:24.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:24.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:22.797698+0000 mgr.smithi045.bvlamo (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.620+0000 7f11dcbab700 1 -- 172.21.15.97:0/3071437495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 msgr2=0x7f11d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.620+0000 7f11dcbab700 1 --2- 172.21.15.97:0/3071437495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d80f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f11c8009fb0 tx=0x7f11c8009b30).stop 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- 172.21.15.97:0/3071437495 shutdown_connections 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 --2- 172.21.15.97:0/3071437495 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d80f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- 172.21.15.97:0/3071437495 >> 172.21.15.97:0/3071437495 conn(0x7f11d80f0b70 msgr2=0x7f11d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- 172.21.15.97:0/3071437495 shutdown_connections 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- 172.21.15.97:0/3071437495 wait complete. 2024-02-19T22:24:26.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 Processor -- start 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- start start 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11dcbab700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d8060220 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.621+0000 7f11d659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11d659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58796/0 (socket says 172.21.15.97:58796) 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11d659c700 1 -- 172.21.15.97:0/1598940720 learned_addr learned my addr 172.21.15.97:0/1598940720 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11d659c700 1 -- 172.21.15.97:0/1598940720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c8005040 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11d659c700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f11c8005010 tx=0x7f11c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c8009320 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11c8013380 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8060420 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11c8014e50 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.622+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8109bb0 con 0x7f11d80f5410 2024-02-19T22:24:26.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.623+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f11c80095f0 con 0x7f11d80f5410 2024-02-19T22:24:26.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.623+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d8059100 con 0x7f11d80f5410 2024-02-19T22:24:26.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.623+0000 7f11c6ffd700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 0x7f11c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:26.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.623+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f11c8040e90 con 0x7f11d80f5410 2024-02-19T22:24:26.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.623+0000 7f11d5d9b700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 0x7f11c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:26.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.624+0000 7f11d5d9b700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 0x7f11c00323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f11cc0096a0 tx=0x7f11cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:26.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.628+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f11c8014420 con 0x7f11d80f5410 2024-02-19T22:24:26.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.834+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f11d8045ad0 con 0x7f11d80f5410 2024-02-19T22:24:26.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.835+0000 7f11c6ffd700 1 -- 172.21.15.97:0/1598940720 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f11c8010230 con 0x7f11d80f5410 2024-02-19T22:24:26.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:26.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:26.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 msgr2=0x7f11c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:26.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 0x7f11c00323d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f11cc0096a0 tx=0x7f11cc006b40).stop 2024-02-19T22:24:26.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 msgr2=0x7f11d8109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:26.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f11c8005010 tx=0x7f11c800b040).stop 2024-02-19T22:24:26.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 shutdown_connections 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f11c002ff10 0x7f11c00323d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 --2- 172.21.15.97:0/1598940720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f11d80f5410 0x7f11d8109500 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 >> 172.21.15.97:0/1598940720 conn(0x7f11d80f0b70 msgr2=0x7f11d80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 shutdown_connections 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:26.837+0000 7f11dcbab700 1 -- 172.21.15.97:0/1598940720 wait complete. 2024-02-19T22:24:26.840 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:26.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:24.797959+0000 mgr.smithi045.bvlamo (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:27.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:27 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:26.836120+0000 mon.smithi045 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.97:0/1598940720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:28.439 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:28.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:28.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:26.798288+0000 mgr.smithi045.bvlamo (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:30.450 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 -- 172.21.15.97:0/4176315347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 msgr2=0x7fe9080f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/4176315347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9080f66e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fe8f0009fb0 tx=0x7fe8f0009b30).stop 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 -- 172.21.15.97:0/4176315347 shutdown_connections 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/4176315347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9080f66e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 -- 172.21.15.97:0/4176315347 >> 172.21.15.97:0/4176315347 conn(0x7fe9080f1ac0 msgr2=0x7fe9080f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 -- 172.21.15.97:0/4176315347 shutdown_connections 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 -- 172.21.15.97:0/4176315347 wait complete. 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.445+0000 7fe90d3f9700 1 Processor -- start 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe90d3f9700 1 -- start start 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe90d3f9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe90d3f9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe908106ad0 con 0x7fe9080f62c0 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe907fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe907fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58804/0 (socket says 172.21.15.97:58804) 2024-02-19T22:24:30.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe907fff700 1 -- 172.21.15.97:0/2692652921 learned_addr learned my addr 172.21.15.97:0/2692652921 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe907fff700 1 -- 172.21.15.97:0/2692652921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f0005040 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.446+0000 7fe907fff700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe8f00089f0 tx=0x7fe8f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8f0009320 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8f0013380 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe908106cd0 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8f0014e20 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe908107160 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8f00095f0 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.447+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe908045ad0 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.448+0000 7fe904ff9700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 0x7fe8f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.448+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe8f0040f70 con 0x7fe9080f62c0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.448+0000 7fe9077fe700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 0x7fe8f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:30.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.448+0000 7fe9077fe700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 0x7fe8f40323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:30.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.452+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8f006d2b0 con 0x7fe9080f62c0 2024-02-19T22:24:30.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.657+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe908108010 con 0x7fe9080f62c0 2024-02-19T22:24:30.660 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.658+0000 7fe904ff9700 1 -- 172.21.15.97:0/2692652921 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8f001d980 con 0x7fe9080f62c0 2024-02-19T22:24:30.661 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:30.661 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:30.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 msgr2=0x7fe8f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:30.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 0x7fe8f40323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fe8f80096a0 tx=0x7fe8f8006b40).stop 2024-02-19T22:24:30.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 msgr2=0x7fe9081065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:30.662 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fe8f00089f0 tx=0x7fe8f000b040).stop 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 shutdown_connections 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe8f402ff10 0x7fe8f40323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 --2- 172.21.15.97:0/2692652921 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe9080f62c0 0x7fe9081065e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 >> 172.21.15.97:0/2692652921 conn(0x7fe9080f1ac0 msgr2=0x7fe9080f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 shutdown_connections 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:30.660+0000 7fe90d3f9700 1 -- 172.21.15.97:0/2692652921 wait complete. 2024-02-19T22:24:30.663 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:28.798665+0000 mgr.smithi045.bvlamo (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:32.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:31 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:30.659268+0000 mon.smithi045 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.97:0/2692652921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:32.488 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:32.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:32.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:30.799028+0000 mgr.smithi045.bvlamo (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 -- 172.21.15.97:0/1146658775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 msgr2=0x7fa5ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 --2- 172.21.15.97:0/1146658775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac0f90f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fa5a4009960 tx=0x7fa5a4004d10).stop 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 -- 172.21.15.97:0/1146658775 shutdown_connections 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 --2- 172.21.15.97:0/1146658775 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac0f90f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 -- 172.21.15.97:0/1146658775 >> 172.21.15.97:0/1146658775 conn(0x7fa5ac0f0b70 msgr2=0x7fa5ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 -- 172.21.15.97:0/1146658775 shutdown_connections 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.648+0000 7fa5b191d700 1 -- 172.21.15.97:0/1146658775 wait complete. 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5b191d700 1 Processor -- start 2024-02-19T22:24:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5b191d700 1 -- start start 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5b191d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5b191d700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ac106ce0 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5aaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5aaffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:58818/0 (socket says 172.21.15.97:58818) 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.649+0000 7fa5aaffd700 1 -- 172.21.15.97:0/3956523570 learned_addr learned my addr 172.21.15.97:0/3956523570 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5aaffd700 1 -- 172.21.15.97:0/3956523570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4009640 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5aaffd700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa5a4007b20 tx=0x7fa5a4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5a4018070 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac106ee0 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa5a4007e60 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5a401d420 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.650+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ac107370 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5a4013630 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ac045ad0 con 0x7fa5ac060280 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5a37fe700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 0x7fa598032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:34.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5a4041590 con 0x7fa5ac060280 2024-02-19T22:24:34.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5aa7fc700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 0x7fa598032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:34.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.651+0000 7fa5aa7fc700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 0x7fa598032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa5940096a0 tx=0x7fa594006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:34.738 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.656+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5a4004810 con 0x7fa5ac060280 2024-02-19T22:24:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.866+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5ac108220 con 0x7fa5ac060280 2024-02-19T22:24:34.868 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.866+0000 7fa5a37fe700 1 -- 172.21.15.97:0/3956523570 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa5a401a2d0 con 0x7fa5ac060280 2024-02-19T22:24:34.869 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:34.869 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 msgr2=0x7fa598032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 0x7fa598032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa5940096a0 tx=0x7fa594006b40).stop 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 msgr2=0x7fa5ac1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fa5a4007b20 tx=0x7fa5a4004d10).stop 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 shutdown_connections 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa59802ff60 0x7fa598032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:34.870 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 --2- 172.21.15.97:0/3956523570 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa5ac060280 0x7fa5ac1067f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 >> 172.21.15.97:0/3956523570 conn(0x7fa5ac0f0b70 msgr2=0x7fa5ac0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 shutdown_connections 2024-02-19T22:24:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:34.868+0000 7fa5b191d700 1 -- 172.21.15.97:0/3956523570 wait complete. 2024-02-19T22:24:34.871 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:35.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:32.799357+0000 mgr.smithi045.bvlamo (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:36.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:35 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:34.867324+0000 mon.smithi045 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.97:0/3956523570' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:36.580 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:36.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:37.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:34.799606+0000 mgr.smithi045.bvlamo (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/2198871804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 msgr2=0x7f6aac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/2198871804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac0f6930 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6a9c009fb0 tx=0x7f6a9c009b30).stop 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/2198871804 shutdown_connections 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/2198871804 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac0f6930 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/2198871804 >> 172.21.15.97:0/2198871804 conn(0x7f6aac0f1c70 msgr2=0x7f6aac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:38.474 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/2198871804 shutdown_connections 2024-02-19T22:24:38.475 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.471+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/2198871804 wait complete. 2024-02-19T22:24:38.478 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab3e3b700 1 Processor -- start 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab3e3b700 1 -- start start 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab3e3b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab3e3b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac108ed0 con 0x7f6aac0f6510 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab1bd7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab1bd7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55434/0 (socket says 172.21.15.97:55434) 2024-02-19T22:24:38.479 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.472+0000 7f6ab1bd7700 1 -- 172.21.15.97:0/3420757036 learned_addr learned my addr 172.21.15.97:0/3420757036 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:38.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6ab1bd7700 1 -- 172.21.15.97:0/3420757036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c005040 con 0x7f6aac0f6510 2024-02-19T22:24:38.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6ab1bd7700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6a9c0089f0 tx=0x7f6a9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:38.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a9c009320 con 0x7f6aac0f6510 2024-02-19T22:24:38.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a9c013380 con 0x7f6aac0f6510 2024-02-19T22:24:38.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac1090d0 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a9c01ed80 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.473+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac109560 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.474+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a9c01e420 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.474+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac045ad0 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.474+0000 7f6aa27fc700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 0x7f6a98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.474+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a9c041110 con 0x7f6aac0f6510 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.474+0000 7f6ab13d6700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 0x7f6a98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:38.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.475+0000 7f6ab13d6700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 0x7f6a98032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6aa80096a0 tx=0x7f6aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:38.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.479+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a9c014c70 con 0x7f6aac0f6510 2024-02-19T22:24:38.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.683+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6aac10a410 con 0x7f6aac0f6510 2024-02-19T22:24:38.685 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.683+0000 7f6aa27fc700 1 -- 172.21.15.97:0/3420757036 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6a9c004490 con 0x7f6aac0f6510 2024-02-19T22:24:38.686 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:38.686 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 msgr2=0x7f6a98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 0x7f6a98032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6aa80096a0 tx=0x7f6aa8006b40).stop 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 msgr2=0x7f6aac1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6a9c0089f0 tx=0x7f6a9c00b040).stop 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 shutdown_connections 2024-02-19T22:24:38.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f6a9802ff60 0x7f6a98032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:38.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 --2- 172.21.15.97:0/3420757036 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f6aac0f6510 0x7f6aac1089e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:38.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 >> 172.21.15.97:0/3420757036 conn(0x7f6aac0f1c70 msgr2=0x7f6aac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:38.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 shutdown_connections 2024-02-19T22:24:38.688 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:38.685+0000 7f6ab3e3b700 1 -- 172.21.15.97:0/3420757036 wait complete. 2024-02-19T22:24:38.688 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:39.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:36.799955+0000 mgr.smithi045.bvlamo (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:40.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:39 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:38.684571+0000 mon.smithi045 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.97:0/3420757036' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:40.470 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:40.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:41.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:38.800221+0000 mgr.smithi045.bvlamo (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 -- 172.21.15.97:0/3147766022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 msgr2=0x7f46c40f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 --2- 172.21.15.97:0/3147766022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c40f5df0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f46b8009fb0 tx=0x7f46b8009b30).stop 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 -- 172.21.15.97:0/3147766022 shutdown_connections 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 --2- 172.21.15.97:0/3147766022 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c40f5df0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 -- 172.21.15.97:0/3147766022 >> 172.21.15.97:0/3147766022 conn(0x7f46c40f02e0 msgr2=0x7f46c40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 -- 172.21.15.97:0/3147766022 shutdown_connections 2024-02-19T22:24:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.563+0000 7f46c9578700 1 -- 172.21.15.97:0/3147766022 wait complete. 2024-02-19T22:24:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c9578700 1 Processor -- start 2024-02-19T22:24:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c9578700 1 -- start start 2024-02-19T22:24:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c9578700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:42.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c9578700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c40f5390 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c2ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:55446/0 (socket says 172.21.15.97:55446) 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c2ffd700 1 -- 172.21.15.97:0/484186196 learned_addr learned my addr 172.21.15.97:0/484186196 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.564+0000 7f46c2ffd700 1 -- 172.21.15.97:0/484186196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b8005040 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46c2ffd700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f46b8005010 tx=0x7f46b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46b8009320 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46b8013380 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c40f5590 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46b8014e50 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c4109ba0 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.565+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46b80263a0 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.566+0000 7f46ab7fe700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 0x7f46ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.566+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46b8040f40 con 0x7f46c40f59d0 2024-02-19T22:24:42.569 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.566+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c4059100 con 0x7f46c40f59d0 2024-02-19T22:24:42.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.566+0000 7f46c27fc700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 0x7f46ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:42.570 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.566+0000 7f46c27fc700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 0x7f46ac0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:42.573 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.571+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46b8012920 con 0x7f46c40f59d0 2024-02-19T22:24:42.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.783+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46c410a1b0 con 0x7f46c40f59d0 2024-02-19T22:24:42.786 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.784+0000 7f46ab7fe700 1 -- 172.21.15.97:0/484186196 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f46b802b020 con 0x7f46c40f59d0 2024-02-19T22:24:42.787 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:42.787 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 msgr2=0x7f46ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 0x7f46ac0323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).stop 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 msgr2=0x7f46c41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f46b8005010 tx=0x7f46b800b040).stop 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 shutdown_connections 2024-02-19T22:24:42.788 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f46ac02ff10 0x7f46ac0323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 --2- 172.21.15.97:0/484186196 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f46c40f59d0 0x7f46c41094f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 >> 172.21.15.97:0/484186196 conn(0x7f46c40f02e0 msgr2=0x7f46c40f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 shutdown_connections 2024-02-19T22:24:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:42.786+0000 7f46c9578700 1 -- 172.21.15.97:0/484186196 wait complete. 2024-02-19T22:24:42.789 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:43.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:40.800522+0000 mgr.smithi045.bvlamo (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:44.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:43 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:42.785097+0000 mon.smithi045 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.97:0/484186196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:44.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:44.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:45.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:42.800906+0000 mgr.smithi045.bvlamo (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:46.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.557+0000 7fa3bd968700 1 -- 172.21.15.97:0/1711870637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 msgr2=0x7fa3b80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.557+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1711870637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b80f4be0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fa3ac009fb0 tx=0x7fa3ac009b30).stop 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.557+0000 7fa3bd968700 1 -- 172.21.15.97:0/1711870637 shutdown_connections 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.557+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1711870637 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b80f4be0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.557+0000 7fa3bd968700 1 -- 172.21.15.97:0/1711870637 >> 172.21.15.97:0/1711870637 conn(0x7fa3b80f02e0 msgr2=0x7fa3b80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 -- 172.21.15.97:0/1711870637 shutdown_connections 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 -- 172.21.15.97:0/1711870637 wait complete. 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 Processor -- start 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 -- start start 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:46.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.558+0000 7fa3bd968700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b8106cc0 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3b6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3b6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48656/0 (socket says 172.21.15.97:48656) 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3b6ffd700 1 -- 172.21.15.97:0/1337364846 learned_addr learned my addr 172.21.15.97:0/1337364846 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3b6ffd700 1 -- 172.21.15.97:0/1337364846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac005040 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3b6ffd700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa3ac005010 tx=0x7fa3ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ac009320 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b8106ec0 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3ac013380 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3ac014e50 con 0x7fa3b80f47c0 2024-02-19T22:24:46.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.559+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b8107350 con 0x7fa3b80f47c0 2024-02-19T22:24:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.560+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3ac014420 con 0x7fa3b80f47c0 2024-02-19T22:24:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.560+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b8045ad0 con 0x7fa3b80f47c0 2024-02-19T22:24:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.560+0000 7fa39f7fe700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 0x7fa3a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.560+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3ac040f40 con 0x7fa3b80f47c0 2024-02-19T22:24:46.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.560+0000 7fa3b67fc700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 0x7fa3a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:46.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.561+0000 7fa3b67fc700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 0x7fa3a00323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fa3a80096a0 tx=0x7fa3a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:46.567 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.565+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3ac026bf0 con 0x7fa3b80f47c0 2024-02-19T22:24:46.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.771+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3b8108200 con 0x7fa3b80f47c0 2024-02-19T22:24:46.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.772+0000 7fa39f7fe700 1 -- 172.21.15.97:0/1337364846 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa3ac01db70 con 0x7fa3b80f47c0 2024-02-19T22:24:46.774 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:46.775 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.773+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 msgr2=0x7fa3a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.773+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 0x7fa3a00323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fa3a80096a0 tx=0x7fa3a8006b40).stop 2024-02-19T22:24:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 msgr2=0x7fa3b81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:46.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fa3ac005010 tx=0x7fa3ac00b040).stop 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 shutdown_connections 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa3a002ff10 0x7fa3a00323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 --2- 172.21.15.97:0/1337364846 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa3b80f47c0 0x7fa3b81067d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 >> 172.21.15.97:0/1337364846 conn(0x7fa3b80f02e0 msgr2=0x7fa3b80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 shutdown_connections 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:46.774+0000 7fa3bd968700 1 -- 172.21.15.97:0/1337364846 wait complete. 2024-02-19T22:24:46.777 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:47.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:44.801308+0000 mgr.smithi045.bvlamo (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:48.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:47 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:46.772770+0000 mon.smithi045 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.97:0/1337364846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:48.402 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:48.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:49.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:46.801670+0000 mgr.smithi045.bvlamo (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- 172.21.15.97:0/2131286465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 msgr2=0x7fc2ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/2131286465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac0f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fc2a0009fb0 tx=0x7fc2a0009b30).stop 2024-02-19T22:24:50.482 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- 172.21.15.97:0/2131286465 shutdown_connections 2024-02-19T22:24:50.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/2131286465 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac0f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:50.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- 172.21.15.97:0/2131286465 >> 172.21.15.97:0/2131286465 conn(0x7fc2ac0f1c70 msgr2=0x7fc2ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:50.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- 172.21.15.97:0/2131286465 shutdown_connections 2024-02-19T22:24:50.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- 172.21.15.97:0/2131286465 wait complete. 2024-02-19T22:24:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 Processor -- start 2024-02-19T22:24:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.480+0000 7fc2b35d3700 1 -- start start 2024-02-19T22:24:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b35d3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:50.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b35d3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ac106cd0 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b136f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b136f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48682/0 (socket says 172.21.15.97:48682) 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b136f700 1 -- 172.21.15.97:0/1641376544 learned_addr learned my addr 172.21.15.97:0/1641376544 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b136f700 1 -- 172.21.15.97:0/1641376544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a0005040 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc2b136f700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc2a0005010 tx=0x7fc2a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.481+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2a0009320 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.482+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ac106ed0 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.482+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2a0013380 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.482+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2a0014e50 con 0x7fc2ac0f6510 2024-02-19T22:24:50.485 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.482+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ac107360 con 0x7fc2ac0f6510 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.483+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ac045ad0 con 0x7fc2ac0f6510 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.483+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2a00095f0 con 0x7fc2ac0f6510 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.483+0000 7fc29dffb700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 0x7fc2980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.483+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2a0040e90 con 0x7fc2ac0f6510 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.483+0000 7fc2b0b6e700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 0x7fc2980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:50.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.484+0000 7fc2b0b6e700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 0x7fc2980323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc2a80096a0 tx=0x7fc2a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:50.490 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.488+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2a0014420 con 0x7fc2ac0f6510 2024-02-19T22:24:50.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.694+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2ac1081b0 con 0x7fc2ac0f6510 2024-02-19T22:24:50.697 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.695+0000 7fc29dffb700 1 -- 172.21.15.97:0/1641376544 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc2a0010230 con 0x7fc2ac0f6510 2024-02-19T22:24:50.697 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:50.698 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:50.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 msgr2=0x7fc2980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:50.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 0x7fc2980323d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fc2a80096a0 tx=0x7fc2a8006b40).stop 2024-02-19T22:24:50.699 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 msgr2=0x7fc2ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fc2a0005010 tx=0x7fc2a000b040).stop 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 shutdown_connections 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc29802ff10 0x7fc2980323d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 --2- 172.21.15.97:0/1641376544 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2ac0f6510 0x7fc2ac1067e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 >> 172.21.15.97:0/1641376544 conn(0x7fc2ac0f1c70 msgr2=0x7fc2ac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 shutdown_connections 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:50.697+0000 7fc2b35d3700 1 -- 172.21.15.97:0/1641376544 wait complete. 2024-02-19T22:24:50.700 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:51.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:48.802013+0000 mgr.smithi045.bvlamo (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:52.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:51 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:50.696133+0000 mon.smithi045 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.97:0/1641376544' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:52.509 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:52.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:53.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:50.802283+0000 mgr.smithi045.bvlamo (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:54.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:52.802575+0000 mgr.smithi045.bvlamo (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 -- 172.21.15.97:0/3684899113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 msgr2=0x7f5bd40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 --2- 172.21.15.97:0/3684899113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd40f55c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f5bd0009fb0 tx=0x7f5bd0009b30).stop 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 -- 172.21.15.97:0/3684899113 shutdown_connections 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 --2- 172.21.15.97:0/3684899113 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd40f55c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 -- 172.21.15.97:0/3684899113 >> 172.21.15.97:0/3684899113 conn(0x7f5bd40f09a0 msgr2=0x7f5bd40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 -- 172.21.15.97:0/3684899113 shutdown_connections 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 -- 172.21.15.97:0/3684899113 wait complete. 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.448+0000 7f5bda188700 1 Processor -- start 2024-02-19T22:24:54.451 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bda188700 1 -- start start 2024-02-19T22:24:54.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bda188700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:54.452 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bda188700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd4106ad0 con 0x7f5bd40f51a0 2024-02-19T22:24:54.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bd9186700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:54.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bd9186700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:48692/0 (socket says 172.21.15.97:48692) 2024-02-19T22:24:54.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bd9186700 1 -- 172.21.15.97:0/1003504698 learned_addr learned my addr 172.21.15.97:0/1003504698 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:54.453 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.449+0000 7f5bd9186700 1 -- 172.21.15.97:0/1003504698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0005040 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bd9186700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5bd00089f0 tx=0x7f5bd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd0009320 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd4106cd0 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5bd0013380 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bd0014e20 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.450+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd4107160 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.451+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd4045ad0 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.451+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5bd00095f0 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.451+0000 7f5bc9ffb700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 0x7f5bc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.451+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5bd0040f50 con 0x7f5bd40f51a0 2024-02-19T22:24:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.451+0000 7f5bd8985700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 0x7f5bc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:54.455 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.452+0000 7f5bd8985700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 0x7f5bc00323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5bc40096a0 tx=0x7f5bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:54.458 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.456+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5bd0014420 con 0x7f5bd40f51a0 2024-02-19T22:24:54.663 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.661+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5bd4108010 con 0x7f5bd40f51a0 2024-02-19T22:24:54.664 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.662+0000 7f5bc9ffb700 1 -- 172.21.15.97:0/1003504698 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5bd001d980 con 0x7f5bd40f51a0 2024-02-19T22:24:54.664 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:54.664 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:54.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 msgr2=0x7f5bc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:54.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 0x7f5bc00323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5bc40096a0 tx=0x7f5bc4006b40).stop 2024-02-19T22:24:54.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 msgr2=0x7f5bd41065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:54.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5bd00089f0 tx=0x7f5bd000b040).stop 2024-02-19T22:24:54.666 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 shutdown_connections 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f5bc002ff10 0x7f5bc00323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 --2- 172.21.15.97:0/1003504698 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f5bd40f51a0 0x7f5bd41065e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 >> 172.21.15.97:0/1003504698 conn(0x7f5bd40f09a0 msgr2=0x7f5bd40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 shutdown_connections 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:54.663+0000 7f5bda188700 1 -- 172.21.15.97:0/1003504698 wait complete. 2024-02-19T22:24:54.667 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:55.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:54.662749+0000 mon.smithi045 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.97:0/1003504698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:56.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:54.802969+0000 mgr.smithi045.bvlamo (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:56.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:56.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:58.521 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- 172.21.15.97:0/1574333236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 msgr2=0x7fccdc0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 --2- 172.21.15.97:0/1574333236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc0f8f20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fccd8009fb0 tx=0x7fccd8009b30).stop 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- 172.21.15.97:0/1574333236 shutdown_connections 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 --2- 172.21.15.97:0/1574333236 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc0f8f20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- 172.21.15.97:0/1574333236 >> 172.21.15.97:0/1574333236 conn(0x7fccdc0f09a0 msgr2=0x7fccdc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- 172.21.15.97:0/1574333236 shutdown_connections 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- 172.21.15.97:0/1574333236 wait complete. 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 Processor -- start 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- start start 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.518+0000 7fcce2851700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccdc108cf0 con 0x7fccdc0601e0 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce184f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:58.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce184f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34004/0 (socket says 172.21.15.97:34004) 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce184f700 1 -- 172.21.15.97:0/3213703599 learned_addr learned my addr 172.21.15.97:0/3213703599 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce184f700 1 -- 172.21.15.97:0/3213703599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8005040 con 0x7fccdc0601e0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce184f700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fccd8005010 tx=0x7fccd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd8009320 con 0x7fccdc0601e0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccdc108ef0 con 0x7fccdc0601e0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccd8013380 con 0x7fccdc0601e0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccd8014e50 con 0x7fccdc0601e0 2024-02-19T22:24:58.523 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.519+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccdc109380 con 0x7fccdc0601e0 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.520+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccd80095f0 con 0x7fccdc0601e0 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.520+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccdc045ad0 con 0x7fccdc0601e0 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.520+0000 7fccd27fc700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 0x7fccc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.520+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccd8040e60 con 0x7fccdc0601e0 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.520+0000 7fcce104e700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 0x7fccc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:58.524 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.521+0000 7fcce104e700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 0x7fccc80323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcccc0096a0 tx=0x7fcccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:58.527 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.525+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccd8014420 con 0x7fccdc0601e0 2024-02-19T22:24:58.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.731+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccdc10a1d0 con 0x7fccdc0601e0 2024-02-19T22:24:58.734 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.732+0000 7fccd27fc700 1 -- 172.21.15.97:0/3213703599 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccd8010230 con 0x7fccdc0601e0 2024-02-19T22:24:58.735 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:24:58.735 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 msgr2=0x7fccc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 0x7fccc80323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcccc0096a0 tx=0x7fcccc006b40).stop 2024-02-19T22:24:58.736 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 msgr2=0x7fccdc108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fccd8005010 tx=0x7fccd800b040).stop 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 shutdown_connections 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fccc802ff10 0x7fccc80323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 --2- 172.21.15.97:0/3213703599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fccdc0601e0 0x7fccdc108800 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 >> 172.21.15.97:0/3213703599 conn(0x7fccdc0f09a0 msgr2=0x7fccdc0f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 shutdown_connections 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:24:58.734+0000 7fcce2851700 1 -- 172.21.15.97:0/3213703599 wait complete. 2024-02-19T22:24:58.737 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:24:58.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:58 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:56.803222+0000 mgr.smithi045.bvlamo (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:59.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:24:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:24:58.733353+0000 mon.smithi045 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.97:0/3213703599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:00.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:00.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:00.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:00 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:24:58.803501+0000 mgr.smithi045.bvlamo (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- 172.21.15.97:0/2679314329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 msgr2=0x7fbdec0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/2679314329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec0f66c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fbde0009fb0 tx=0x7fbde0009b30).stop 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- 172.21.15.97:0/2679314329 shutdown_connections 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/2679314329 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec0f66c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- 172.21.15.97:0/2679314329 >> 172.21.15.97:0/2679314329 conn(0x7fbdec0f1aa0 msgr2=0x7fbdec0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:02.332 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- 172.21.15.97:0/2679314329 shutdown_connections 2024-02-19T22:25:02.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- 172.21.15.97:0/2679314329 wait complete. 2024-02-19T22:25:02.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 Processor -- start 2024-02-19T22:25:02.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.329+0000 7fbdf1f89700 1 -- start start 2024-02-19T22:25:02.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf1f89700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:02.333 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf1f89700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec069e20 con 0x7fbdec0f62a0 2024-02-19T22:25:02.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf0f87700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:02.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf0f87700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34010/0 (socket says 172.21.15.97:34010) 2024-02-19T22:25:02.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf0f87700 1 -- 172.21.15.97:0/3203475291 learned_addr learned my addr 172.21.15.97:0/3203475291 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:02.334 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.330+0000 7fbdf0f87700 1 -- 172.21.15.97:0/3203475291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0005040 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbdf0f87700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fbde0009830 tx=0x7fbde000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde0009320 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec069fc0 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbde0013380 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbde0014420 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.331+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec06a450 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbde001e420 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec059100 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbde97fa700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbde0040ee0 con 0x7fbdec0f62a0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbdebfff700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:02.335 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.332+0000 7fbdebfff700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 0x7fbddc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbdd80096a0 tx=0x7fbdd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:02.339 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.337+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbde001c970 con 0x7fbdec0f62a0 2024-02-19T22:25:02.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.540+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbdec045ad0 con 0x7fbdec0f62a0 2024-02-19T22:25:02.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.541+0000 7fbde97fa700 1 -- 172.21.15.97:0/3203475291 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbde0004490 con 0x7fbdec0f62a0 2024-02-19T22:25:02.543 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:02.543 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:02.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 msgr2=0x7fbddc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:02.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 0x7fbddc032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fbdd80096a0 tx=0x7fbdd8006b40).stop 2024-02-19T22:25:02.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 msgr2=0x7fbdec06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:02.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fbde0009830 tx=0x7fbde000b040).stop 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 shutdown_connections 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.542+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbddc02ff60 0x7fbddc032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.543+0000 7fbdf1f89700 1 --2- 172.21.15.97:0/3203475291 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbdec0f62a0 0x7fbdec06b9a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.543+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 >> 172.21.15.97:0/3203475291 conn(0x7fbdec0f1aa0 msgr2=0x7fbdec0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.543+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 shutdown_connections 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:02.543+0000 7fbdf1f89700 1 -- 172.21.15.97:0/3203475291 wait complete. 2024-02-19T22:25:02.546 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:02.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:00.803802+0000 mgr.smithi045.bvlamo (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:02.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:02 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:02.542014+0000 mon.smithi045 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.97:0/3203475291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:04.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:04.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:02.804177+0000 mgr.smithi045.bvlamo (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:03.940885+0000 mon.smithi045 (mon.0) 314 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:25:04.892 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:03.941134+0000 mon.smithi045 (mon.0) 315 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- 172.21.15.97:0/3703602772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 msgr2=0x7fc2000f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3703602772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc2000f66c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fc1fc009fb0 tx=0x7fc1fc009b30).stop 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- 172.21.15.97:0/3703602772 shutdown_connections 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3703602772 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc2000f66c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- 172.21.15.97:0/3703602772 >> 172.21.15.97:0/3703602772 conn(0x7fc2000f1aa0 msgr2=0x7fc2000f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- 172.21.15.97:0/3703602772 shutdown_connections 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- 172.21.15.97:0/3703602772 wait complete. 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 Processor -- start 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- start start 2024-02-19T22:25:06.273 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.269+0000 7fc2066c2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc200069e20 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2056c0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2056c0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:34036/0 (socket says 172.21.15.97:34036) 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2056c0700 1 -- 172.21.15.97:0/3859150301 learned_addr learned my addr 172.21.15.97:0/3859150301 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2056c0700 1 -- 172.21.15.97:0/3859150301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc005040 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2056c0700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc1fc0089f0 tx=0x7fc1fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1fc009320 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc200069fc0 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1fc013380 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1fc014e20 con 0x7fc2000f62a0 2024-02-19T22:25:06.274 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.270+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20006a450 con 0x7fc2000f62a0 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.271+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1fc014420 con 0x7fc2000f62a0 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.271+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc200059100 con 0x7fc2000f62a0 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.271+0000 7fc1f5ffb700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 0x7fc1ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.271+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1fc040fb0 con 0x7fc2000f62a0 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.272+0000 7fc204ebf700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 0x7fc1ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.272+0000 7fc204ebf700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 0x7fc1ec0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc1f00096a0 tx=0x7fc1f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:06.278 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.276+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1fc014790 con 0x7fc2000f62a0 2024-02-19T22:25:06.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.484+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2000fbe30 con 0x7fc2000f62a0 2024-02-19T22:25:06.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.485+0000 7fc1f5ffb700 1 -- 172.21.15.97:0/3859150301 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1fc01d7f0 con 0x7fc2000f62a0 2024-02-19T22:25:06.487 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:06.487 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 msgr2=0x7fc1ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 0x7fc1ec0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7fc1f00096a0 tx=0x7fc1f0006b40).stop 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 msgr2=0x7fc20006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc1fc0089f0 tx=0x7fc1fc00b040).stop 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 shutdown_connections 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc1ec02ff10 0x7fc1ec0323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 --2- 172.21.15.97:0/3859150301 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc2000f62a0 0x7fc20006b9a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 >> 172.21.15.97:0/3859150301 conn(0x7fc2000f1aa0 msgr2=0x7fc2000f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 shutdown_connections 2024-02-19T22:25:06.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:06.486+0000 7fc2066c2700 1 -- 172.21.15.97:0/3859150301 wait complete. 2024-02-19T22:25:06.490 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:07.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:04.804555+0000 mgr.smithi045.bvlamo (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:07.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:06.485747+0000 mon.smithi045 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.97:0/3859150301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:08.141 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:08.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:09.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:06.804910+0000 mgr.smithi045.bvlamo (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/790884362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 msgr2=0x7fb9e40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/790884362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e40f55c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fb9d8009fb0 tx=0x7fb9d8009b30).stop 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/790884362 shutdown_connections 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/790884362 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e40f55c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/790884362 >> 172.21.15.97:0/790884362 conn(0x7fb9e40f09a0 msgr2=0x7fb9e40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/790884362 shutdown_connections 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/790884362 wait complete. 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 Processor -- start 2024-02-19T22:25:10.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.189+0000 7fb9e9b1c700 1 -- start start 2024-02-19T22:25:10.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e9b1c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:10.193 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e9b1c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e4106ac0 con 0x7fb9e40f51a0 2024-02-19T22:25:10.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e8b1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:10.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e8b1a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:60176/0 (socket says 172.21.15.97:60176) 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e8b1a700 1 -- 172.21.15.97:0/3938363595 learned_addr learned my addr 172.21.15.97:0/3938363595 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e8b1a700 1 -- 172.21.15.97:0/3938363595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.190+0000 7fb9e8b1a700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb9d80089f0 tx=0x7fb9d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d8009320 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e4106cc0 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9d8013380 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d8014e20 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e4107150 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.191+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb9d80263a0 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.192+0000 7fb9e17fa700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 0x7fb9d40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.192+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb9d8040f30 con 0x7fb9e40f51a0 2024-02-19T22:25:10.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.192+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e4045ad0 con 0x7fb9e40f51a0 2024-02-19T22:25:10.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.192+0000 7fb9e3fff700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 0x7fb9d40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:10.196 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.192+0000 7fb9e3fff700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 0x7fb9d40323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb9d00096a0 tx=0x7fb9d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:10.199 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.197+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9d8012920 con 0x7fb9e40f51a0 2024-02-19T22:25:10.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.405+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9e41000e0 con 0x7fb9e40f51a0 2024-02-19T22:25:10.408 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.407+0000 7fb9e17fa700 1 -- 172.21.15.97:0/3938363595 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9d802b020 con 0x7fb9e40f51a0 2024-02-19T22:25:10.409 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:10.409 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.407+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 msgr2=0x7fb9d40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.407+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 0x7fb9d40323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb9d00096a0 tx=0x7fb9d0006b40).stop 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 msgr2=0x7fb9e41065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb9d80089f0 tx=0x7fb9d800b040).stop 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 shutdown_connections 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fb9d402ff10 0x7fb9d40323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:10.410 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 --2- 172.21.15.97:0/3938363595 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fb9e40f51a0 0x7fb9e41065d0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 >> 172.21.15.97:0/3938363595 conn(0x7fb9e40f09a0 msgr2=0x7fb9e405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 shutdown_connections 2024-02-19T22:25:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:10.408+0000 7fb9e9b1c700 1 -- 172.21.15.97:0/3938363595 wait complete. 2024-02-19T22:25:10.411 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:11.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:08.805273+0000 mgr.smithi045.bvlamo (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:11.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:10.406995+0000 mon.smithi045 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.97:0/3938363595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:12.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:12.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:13.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:12 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:10.805676+0000 mgr.smithi045.bvlamo (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 -- 172.21.15.97:0/560725581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 msgr2=0x7f9ed00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/560725581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed00f6930 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f9ec4009fb0 tx=0x7f9ec4009b30).stop 2024-02-19T22:25:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 -- 172.21.15.97:0/560725581 shutdown_connections 2024-02-19T22:25:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/560725581 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed00f6930 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:14.033 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 -- 172.21.15.97:0/560725581 >> 172.21.15.97:0/560725581 conn(0x7f9ed00f1c70 msgr2=0x7f9ed00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 -- 172.21.15.97:0/560725581 shutdown_connections 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.992+0000 7f9ed6d80700 1 -- 172.21.15.97:0/560725581 wait complete. 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.993+0000 7f9ed6d80700 1 Processor -- start 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.993+0000 7f9ed6d80700 1 -- start start 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.993+0000 7f9ed6d80700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.993+0000 7f9ed6d80700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed0106cd0 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed4b1c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed4b1c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:60202/0 (socket says 172.21.15.97:60202) 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed4b1c700 1 -- 172.21.15.97:0/1109122891 learned_addr learned my addr 172.21.15.97:0/1109122891 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed4b1c700 1 -- 172.21.15.97:0/1109122891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4005040 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed4b1c700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f9ec4005010 tx=0x7f9ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4009320 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed0106ed0 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ec4013380 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.994+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ec4014e50 con 0x7f9ed00f6510 2024-02-19T22:25:14.034 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.995+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed0107360 con 0x7f9ed00f6510 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.995+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ec40095f0 con 0x7f9ed00f6510 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.995+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed0045ad0 con 0x7f9ed00f6510 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.996+0000 7f9ecd7fa700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 0x7f9ec00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.996+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ec4040e90 con 0x7f9ed00f6510 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.996+0000 7f9ecffff700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 0x7f9ec00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:13.996+0000 7f9ecffff700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 0x7f9ec00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:14.035 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.000+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ec4014420 con 0x7f9ed00f6510 2024-02-19T22:25:14.207 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.205+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ed0108210 con 0x7f9ed00f6510 2024-02-19T22:25:14.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.206+0000 7f9ecd7fa700 1 -- 172.21.15.97:0/1109122891 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ec4010230 con 0x7f9ed00f6510 2024-02-19T22:25:14.208 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:14.208 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 msgr2=0x7f9ec00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 0x7f9ec00323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f9ebc0096a0 tx=0x7f9ebc006b40).stop 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 msgr2=0x7f9ed01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f9ec4005010 tx=0x7f9ec400b040).stop 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 shutdown_connections 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9ec002ff10 0x7f9ec00323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 --2- 172.21.15.97:0/1109122891 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9ed00f6510 0x7f9ed01067e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 >> 172.21.15.97:0/1109122891 conn(0x7f9ed00f1c70 msgr2=0x7f9ed00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 shutdown_connections 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:14.207+0000 7f9ed6d80700 1 -- 172.21.15.97:0/1109122891 wait complete. 2024-02-19T22:25:14.210 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:15.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:12.806092+0000 mgr.smithi045.bvlamo (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:15.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:14.206626+0000 mon.smithi045 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.97:0/1109122891' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:15.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:15.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:14.806438+0000 mgr.smithi045.bvlamo (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/227029853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 msgr2=0x7fc7bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/227029853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc0f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fc7b0009fb0 tx=0x7fc7b0009b30).stop 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/227029853 shutdown_connections 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/227029853 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc0f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/227029853 >> 172.21.15.97:0/227029853 conn(0x7fc7bc0f0b70 msgr2=0x7fc7bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:17.839 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/227029853 shutdown_connections 2024-02-19T22:25:17.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/227029853 wait complete. 2024-02-19T22:25:17.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.836+0000 7fc7c2a9e700 1 Processor -- start 2024-02-19T22:25:17.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c2a9e700 1 -- start start 2024-02-19T22:25:17.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c2a9e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:17.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c2a9e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc106cd0 con 0x7fc7bc0f5410 2024-02-19T22:25:17.841 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c083a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c083a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37366/0 (socket says 172.21.15.97:37366) 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.837+0000 7fc7c083a700 1 -- 172.21.15.97:0/909476796 learned_addr learned my addr 172.21.15.97:0/909476796 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7c083a700 1 -- 172.21.15.97:0/909476796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b0005040 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7c083a700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc7b0005010 tx=0x7fc7b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7b0009320 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7b0013380 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc106ed0 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7b0014e50 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.838+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc107360 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7b00095f0 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc045ad0 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7b97fa700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 0x7fc7ac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7b0040fa0 con 0x7fc7bc0f5410 2024-02-19T22:25:17.842 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7bbfff700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 0x7fc7ac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:17.843 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.839+0000 7fc7bbfff700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 0x7fc7ac0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc7a80096a0 tx=0x7fc7a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:17.846 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:17.844+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7b0014420 con 0x7fc7bc0f5410 2024-02-19T22:25:18.049 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.047+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7bc0fafa0 con 0x7fc7bc0f5410 2024-02-19T22:25:18.050 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.049+0000 7fc7b97fa700 1 -- 172.21.15.97:0/909476796 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc7b0010230 con 0x7fc7bc0f5410 2024-02-19T22:25:18.050 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:18.051 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 msgr2=0x7fc7ac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 0x7fc7ac0323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fc7a80096a0 tx=0x7fc7a8006b40).stop 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 msgr2=0x7fc7bc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fc7b0005010 tx=0x7fc7b000b040).stop 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 shutdown_connections 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fc7ac02ff10 0x7fc7ac0323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 --2- 172.21.15.97:0/909476796 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fc7bc0f5410 0x7fc7bc1067e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:18.052 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 >> 172.21.15.97:0/909476796 conn(0x7fc7bc0f0b70 msgr2=0x7fc7bc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:18.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 shutdown_connections 2024-02-19T22:25:18.053 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:18.050+0000 7fc7c2a9e700 1 -- 172.21.15.97:0/909476796 wait complete. 2024-02-19T22:25:18.053 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:19.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:16.806790+0000 mgr.smithi045.bvlamo (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:19.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:18.049119+0000 mon.smithi045 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.97:0/909476796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:19.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:19.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:21.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:18.807168+0000 mgr.smithi045.bvlamo (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:21.700 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.697+0000 7fac8cbe2700 1 -- 172.21.15.97:0/3014093933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 msgr2=0x7fac880f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.697+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/3014093933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac880f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fac70009fb0 tx=0x7fac70009b30).stop 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 -- 172.21.15.97:0/3014093933 shutdown_connections 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/3014093933 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac880f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 -- 172.21.15.97:0/3014093933 >> 172.21.15.97:0/3014093933 conn(0x7fac880f1c70 msgr2=0x7fac880f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 -- 172.21.15.97:0/3014093933 shutdown_connections 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 -- 172.21.15.97:0/3014093933 wait complete. 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 Processor -- start 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.698+0000 7fac8cbe2700 1 -- start start 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8cbe2700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:21.701 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8cbe2700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88106cd0 con 0x7fac880f6510 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8659c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37380/0 (socket says 172.21.15.97:37380) 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8659c700 1 -- 172.21.15.97:0/174639040 learned_addr learned my addr 172.21.15.97:0/174639040 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8659c700 1 -- 172.21.15.97:0/174639040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70005040 con 0x7fac880f6510 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8659c700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fac700089f0 tx=0x7fac7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:21.703 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac70009320 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac70013380 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac88106ed0 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac7001ed80 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.699+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac88107360 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.700+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac7001e420 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.700+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac88045ad0 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.700+0000 7fac7effd700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.700+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac70041150 con 0x7fac880f6510 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.701+0000 7fac85d9b700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:21.704 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.701+0000 7fac85d9b700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 0x7fac74032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fac780096a0 tx=0x7fac78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:21.707 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.706+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac700095f0 con 0x7fac880f6510 2024-02-19T22:25:21.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.913+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac88108210 con 0x7fac880f6510 2024-02-19T22:25:21.915 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.913+0000 7fac7effd700 1 -- 172.21.15.97:0/174639040 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fac70004490 con 0x7fac880f6510 2024-02-19T22:25:21.916 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:21.916 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:21.917 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 msgr2=0x7fac74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 0x7fac74032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fac780096a0 tx=0x7fac78006b40).stop 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 msgr2=0x7fac881067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fac700089f0 tx=0x7fac7000b040).stop 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 shutdown_connections 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fac7402ff60 0x7fac74032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 --2- 172.21.15.97:0/174639040 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fac880f6510 0x7fac881067e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 >> 172.21.15.97:0/174639040 conn(0x7fac880f1c70 msgr2=0x7fac880f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 shutdown_connections 2024-02-19T22:25:21.918 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:21.915+0000 7fac8cbe2700 1 -- 172.21.15.97:0/174639040 wait complete. 2024-02-19T22:25:21.919 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:23.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:20.807564+0000 mgr.smithi045.bvlamo (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:23.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:21.914612+0000 mon.smithi045 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.97:0/174639040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:23.560 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:23.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:25.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:22.807949+0000 mgr.smithi045.bvlamo (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- 172.21.15.97:0/3936134511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 msgr2=0x7faf3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 --2- 172.21.15.97:0/3936134511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7faf30009fb0 tx=0x7faf30009b30).stop 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- 172.21.15.97:0/3936134511 shutdown_connections 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 --2- 172.21.15.97:0/3936134511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- 172.21.15.97:0/3936134511 >> 172.21.15.97:0/3936134511 conn(0x7faf3c0f0b70 msgr2=0x7faf3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- 172.21.15.97:0/3936134511 shutdown_connections 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- 172.21.15.97:0/3936134511 wait complete. 2024-02-19T22:25:25.518 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 Processor -- start 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- start start 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.513+0000 7faf44444700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf3c105bd0 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf421e0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf421e0700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:37396/0 (socket says 172.21.15.97:37396) 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf421e0700 1 -- 172.21.15.97:0/258403684 learned_addr learned my addr 172.21.15.97:0/258403684 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf421e0700 1 -- 172.21.15.97:0/258403684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf30005040 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf421e0700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7faf30005010 tx=0x7faf3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30009320 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf3c105dd0 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf30013380 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf30014e50 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.514+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf3c106260 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.515+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf300095f0 con 0x7faf3c0f5410 2024-02-19T22:25:25.519 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.515+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf3c045ad0 con 0x7faf3c0f5410 2024-02-19T22:25:25.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.515+0000 7faf2effd700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 0x7faf280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:25.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.515+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf30040fa0 con 0x7faf3c0f5410 2024-02-19T22:25:25.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.515+0000 7faf419df700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 0x7faf280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:25.520 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.516+0000 7faf419df700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 0x7faf280323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7faf380096a0 tx=0x7faf38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:25.522 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.520+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf30014420 con 0x7faf3c0f5410 2024-02-19T22:25:25.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.725+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf3c107110 con 0x7faf3c0f5410 2024-02-19T22:25:25.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.726+0000 7faf2effd700 1 -- 172.21.15.97:0/258403684 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faf30010230 con 0x7faf3c0f5410 2024-02-19T22:25:25.729 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:25.729 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:25.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 msgr2=0x7faf280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:25.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 0x7faf280323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7faf380096a0 tx=0x7faf38006b40).stop 2024-02-19T22:25:25.730 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 msgr2=0x7faf3c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7faf30005010 tx=0x7faf3000b040).stop 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 shutdown_connections 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7faf2802ff10 0x7faf280323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 --2- 172.21.15.97:0/258403684 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7faf3c0f5410 0x7faf3c1056e0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 >> 172.21.15.97:0/258403684 conn(0x7faf3c0f0b70 msgr2=0x7faf3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 shutdown_connections 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:25.728+0000 7faf44444700 1 -- 172.21.15.97:0/258403684 wait complete. 2024-02-19T22:25:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:27.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:24.808337+0000 mgr.smithi045.bvlamo (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:27.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:25.727275+0000 mon.smithi045 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.97:0/258403684' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:27.380 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:27.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:29.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:26.808663+0000 mgr.smithi045.bvlamo (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 -- 172.21.15.97:0/3323869381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 msgr2=0x7f89f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 --2- 172.21.15.97:0/3323869381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f80f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f89ec009fb0 tx=0x7f89ec009b30).stop 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 -- 172.21.15.97:0/3323869381 shutdown_connections 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 --2- 172.21.15.97:0/3323869381 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f80f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 -- 172.21.15.97:0/3323869381 >> 172.21.15.97:0/3323869381 conn(0x7f89f80f0b70 msgr2=0x7f89f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 -- 172.21.15.97:0/3323869381 shutdown_connections 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.507+0000 7f89fd76b700 1 -- 172.21.15.97:0/3323869381 wait complete. 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89fd76b700 1 Processor -- start 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89fd76b700 1 -- start start 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89fd76b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89fd76b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89f8106ca0 con 0x7f89f80f5410 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89f6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:29.529 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89f6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:40780/0 (socket says 172.21.15.97:40780) 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.508+0000 7f89f6ffd700 1 -- 172.21.15.97:0/2690908918 learned_addr learned my addr 172.21.15.97:0/2690908918 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89f6ffd700 1 -- 172.21.15.97:0/2690908918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec005040 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89f6ffd700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f89ec005010 tx=0x7f89ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89ec009320 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f8106ea0 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89ec013380 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89ec014e50 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.509+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f8107330 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.510+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89ec0095f0 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.510+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89f8045ad0 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.510+0000 7f89df7fe700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 0x7f89e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.510+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89ec040f80 con 0x7f89f80f5410 2024-02-19T22:25:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.510+0000 7f89f67fc700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 0x7f89e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.511+0000 7f89f67fc700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 0x7f89e00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f89e80096a0 tx=0x7f89e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:29.531 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.515+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89ec014420 con 0x7f89f80f5410 2024-02-19T22:25:29.725 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.723+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89f81081e0 con 0x7f89f80f5410 2024-02-19T22:25:29.726 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.723+0000 7f89df7fe700 1 -- 172.21.15.97:0/2690908918 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f89ec010230 con 0x7f89f80f5410 2024-02-19T22:25:29.726 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:29.726 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 msgr2=0x7f89e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 0x7f89e00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f89e80096a0 tx=0x7f89e8006b40).stop 2024-02-19T22:25:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 msgr2=0x7f89f81067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f89ec005010 tx=0x7f89ec00b040).stop 2024-02-19T22:25:29.727 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 shutdown_connections 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f89e002ff10 0x7f89e00323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 --2- 172.21.15.97:0/2690908918 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f89f80f5410 0x7f89f81067b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 >> 172.21.15.97:0/2690908918 conn(0x7f89f80f0b70 msgr2=0x7f89f805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 shutdown_connections 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:29.725+0000 7f89fd76b700 1 -- 172.21.15.97:0/2690908918 wait complete. 2024-02-19T22:25:29.728 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:28.808960+0000 mgr.smithi045.bvlamo (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:29.724502+0000 mon.smithi045 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.97:0/2690908918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:31.444 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:31.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:33.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:30.809323+0000 mgr.smithi045.bvlamo (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:33.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 -- 172.21.15.97:0/3348097014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 msgr2=0x7f461c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:33.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 --2- 172.21.15.97:0/3348097014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c0f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4610009fb0 tx=0x7f4610009b30).stop 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 -- 172.21.15.97:0/3348097014 shutdown_connections 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 --2- 172.21.15.97:0/3348097014 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c0f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 -- 172.21.15.97:0/3348097014 >> 172.21.15.97:0/3348097014 conn(0x7f461c0f0b70 msgr2=0x7f461c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 -- 172.21.15.97:0/3348097014 shutdown_connections 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.430+0000 7f46220bb700 1 -- 172.21.15.97:0/3348097014 wait complete. 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f46220bb700 1 Processor -- start 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f46220bb700 1 -- start start 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f46220bb700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:33.434 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f46220bb700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c105bd0 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f461b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:40810/0 (socket says 172.21.15.97:40810) 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.431+0000 7f461b7fe700 1 -- 172.21.15.97:0/1534294218 learned_addr learned my addr 172.21.15.97:0/1534294218 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f461b7fe700 1 -- 172.21.15.97:0/1534294218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610005040 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f461b7fe700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4610005010 tx=0x7f461000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4610009320 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c105dd0 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4610013380 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4610014e50 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.432+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c106260 con 0x7f461c0f5410 2024-02-19T22:25:33.436 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.433+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46100095f0 con 0x7f461c0f5410 2024-02-19T22:25:33.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.433+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c045ad0 con 0x7f461c0f5410 2024-02-19T22:25:33.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.433+0000 7f4603fff700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 0x7f46040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:33.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.433+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4610040fa0 con 0x7f461c0f5410 2024-02-19T22:25:33.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.433+0000 7f461affd700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 0x7f46040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:33.437 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.434+0000 7f461affd700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 0x7f46040323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f460c0096a0 tx=0x7f460c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:33.440 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.438+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4610014420 con 0x7f461c0f5410 2024-02-19T22:25:33.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.645+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f461c107110 con 0x7f461c0f5410 2024-02-19T22:25:33.647 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.645+0000 7f4603fff700 1 -- 172.21.15.97:0/1534294218 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4610010230 con 0x7f461c0f5410 2024-02-19T22:25:33.648 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:33.648 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:33.649 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 msgr2=0x7f46040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 0x7f46040323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f460c0096a0 tx=0x7f460c006b40).stop 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 msgr2=0x7f461c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4610005010 tx=0x7f461000b040).stop 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 shutdown_connections 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f460402ff10 0x7f46040323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 --2- 172.21.15.97:0/1534294218 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f461c0f5410 0x7f461c1056e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 >> 172.21.15.97:0/1534294218 conn(0x7f461c0f0b70 msgr2=0x7f461c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 shutdown_connections 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:33.647+0000 7f46220bb700 1 -- 172.21.15.97:0/1534294218 wait complete. 2024-02-19T22:25:33.650 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:34.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:33.646477+0000 mon.smithi045 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.97:0/1534294218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:35.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:32.809637+0000 mgr.smithi045.bvlamo (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:35.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:35.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:37.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:34.809989+0000 mgr.smithi045.bvlamo (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:37.466 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1692169038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 msgr2=0x7fbe000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1692169038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe000f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbdf4009fb0 tx=0x7fbdf4009b30).stop 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1692169038 shutdown_connections 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1692169038 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe000f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1692169038 >> 172.21.15.97:0/1692169038 conn(0x7fbe000f1c70 msgr2=0x7fbe000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1692169038 shutdown_connections 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.462+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1692169038 wait complete. 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbe05b4b700 1 Processor -- start 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbe05b4b700 1 -- start start 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbe05b4b700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbe05b4b700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00069e20 con 0x7fbe000f6510 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbdff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:37.467 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbdff7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53746/0 (socket says 172.21.15.97:53746) 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.463+0000 7fbdff7fe700 1 -- 172.21.15.97:0/1787500092 learned_addr learned my addr 172.21.15.97:0/1787500092 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbdff7fe700 1 -- 172.21.15.97:0/1787500092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4005040 con 0x7fbe000f6510 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbdff7fe700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbdf40089f0 tx=0x7fbdf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4009320 con 0x7fbe000f6510 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe00069fc0 con 0x7fbe000f6510 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbdf4013380 con 0x7fbe000f6510 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf401ed80 con 0x7fbe000f6510 2024-02-19T22:25:37.468 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.464+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0006a450 con 0x7fbe000f6510 2024-02-19T22:25:37.469 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.465+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbdf401e420 con 0x7fbe000f6510 2024-02-19T22:25:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.465+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00059100 con 0x7fbe000f6510 2024-02-19T22:25:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.465+0000 7fbde7fff700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 0x7fbde8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.465+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbdf4040fe0 con 0x7fbe000f6510 2024-02-19T22:25:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.465+0000 7fbdfeffd700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 0x7fbde8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:37.470 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.466+0000 7fbdfeffd700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 0x7fbde8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbdf00096a0 tx=0x7fbdf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:37.472 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.470+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdf40095f0 con 0x7fbe000f6510 2024-02-19T22:25:37.680 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.678+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe00045ad0 con 0x7fbe000f6510 2024-02-19T22:25:37.681 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.678+0000 7fbde7fff700 1 -- 172.21.15.97:0/1787500092 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbdf401cb10 con 0x7fbe000f6510 2024-02-19T22:25:37.681 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:37.681 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:37.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 msgr2=0x7fbde8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:37.682 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 0x7fbde8032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbdf00096a0 tx=0x7fbdf0006b40).stop 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 msgr2=0x7fbe0006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbdf40089f0 tx=0x7fbdf400b040).stop 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 shutdown_connections 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbde802ff60 0x7fbde8032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 --2- 172.21.15.97:0/1787500092 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbe000f6510 0x7fbe0006b9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 >> 172.21.15.97:0/1787500092 conn(0x7fbe000f1c70 msgr2=0x7fbe000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 shutdown_connections 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:37.680+0000 7fbe05b4b700 1 -- 172.21.15.97:0/1787500092 wait complete. 2024-02-19T22:25:37.683 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:39.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:36.810288+0000 mgr.smithi045.bvlamo (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:39.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:38 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:37.679457+0000 mon.smithi045 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.97:0/1787500092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:39.385 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:39.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:41.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:40 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:38.810637+0000 mgr.smithi045.bvlamo (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:41.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- 172.21.15.97:0/3093973508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 msgr2=0x7fe1280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:41.428 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/3093973508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe1280f6930 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe118009fb0 tx=0x7fe118009b30).stop 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- 172.21.15.97:0/3093973508 shutdown_connections 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/3093973508 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe1280f6930 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- 172.21.15.97:0/3093973508 >> 172.21.15.97:0/3093973508 conn(0x7fe1280f1c70 msgr2=0x7fe1280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- 172.21.15.97:0/3093973508 shutdown_connections 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- 172.21.15.97:0/3093973508 wait complete. 2024-02-19T22:25:41.429 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 Processor -- start 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.426+0000 7fe12e4a3700 1 -- start start 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe12e4a3700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe12e4a3700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128069e20 con 0x7fe1280f6510 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe127fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe127fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53762/0 (socket says 172.21.15.97:53762) 2024-02-19T22:25:41.431 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe127fff700 1 -- 172.21.15.97:0/2332167167 learned_addr learned my addr 172.21.15.97:0/2332167167 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe127fff700 1 -- 172.21.15.97:0/2332167167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe118005040 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe127fff700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe118005010 tx=0x7fe11800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe118009320 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128069fc0 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe118013380 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe118014e50 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.427+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12806a450 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.428+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1180095f0 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.428+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe128059100 con 0x7fe1280f6510 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.428+0000 7fe124ff9700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 0x7fe1100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:41.432 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.428+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe118040e90 con 0x7fe1280f6510 2024-02-19T22:25:41.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.429+0000 7fe1277fe700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 0x7fe1100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:41.433 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.429+0000 7fe1277fe700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 0x7fe1100323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe11c0096a0 tx=0x7fe11c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:41.435 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.433+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe118014420 con 0x7fe1280f6510 2024-02-19T22:25:41.639 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.637+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1280fc0a0 con 0x7fe1280f6510 2024-02-19T22:25:41.640 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.638+0000 7fe124ff9700 1 -- 172.21.15.97:0/2332167167 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe118010230 con 0x7fe1280f6510 2024-02-19T22:25:41.641 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:41.641 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:41.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 msgr2=0x7fe1100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:41.642 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 0x7fe1100323d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe11c0096a0 tx=0x7fe11c006b40).stop 2024-02-19T22:25:41.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 msgr2=0x7fe12806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:41.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fe118005010 tx=0x7fe11800b040).stop 2024-02-19T22:25:41.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 shutdown_connections 2024-02-19T22:25:41.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fe11002ff10 0x7fe1100323d0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:41.643 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 --2- 172.21.15.97:0/2332167167 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fe1280f6510 0x7fe12806b9a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:41.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 >> 172.21.15.97:0/2332167167 conn(0x7fe1280f1c70 msgr2=0x7fe1280f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:41.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 shutdown_connections 2024-02-19T22:25:41.644 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:41.639+0000 7fe12e4a3700 1 -- 172.21.15.97:0/2332167167 wait complete. 2024-02-19T22:25:41.644 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:42.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:41 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:41.638742+0000 mon.smithi045 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.97:0/2332167167' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:43.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:42 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:40.810947+0000 mgr.smithi045.bvlamo (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:43.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:43.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:45.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:44 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:42.811318+0000 mgr.smithi045.bvlamo (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:45.542 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 -- 172.21.15.97:0/2197312431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 msgr2=0x7fa70c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 --2- 172.21.15.97:0/2197312431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c0f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fa6fc009fb0 tx=0x7fa6fc009b30).stop 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 -- 172.21.15.97:0/2197312431 shutdown_connections 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 --2- 172.21.15.97:0/2197312431 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c0f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 -- 172.21.15.97:0/2197312431 >> 172.21.15.97:0/2197312431 conn(0x7fa70c0f1c70 msgr2=0x7fa70c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 -- 172.21.15.97:0/2197312431 shutdown_connections 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.538+0000 7fa712041700 1 -- 172.21.15.97:0/2197312431 wait complete. 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.539+0000 7fa712041700 1 Processor -- start 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.539+0000 7fa712041700 1 -- start start 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.539+0000 7fa712041700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.539+0000 7fa712041700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa70c106cd0 con 0x7fa70c0f6510 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa70b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa70b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53780/0 (socket says 172.21.15.97:53780) 2024-02-19T22:25:45.543 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa70b7fe700 1 -- 172.21.15.97:0/3887873564 learned_addr learned my addr 172.21.15.97:0/3887873564 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa70b7fe700 1 -- 172.21.15.97:0/3887873564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc005040 con 0x7fa70c0f6510 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa70b7fe700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa6fc005010 tx=0x7fa6fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6fc009320 con 0x7fa70c0f6510 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa70c106ed0 con 0x7fa70c0f6510 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa6fc013380 con 0x7fa70c0f6510 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6fc014e50 con 0x7fa70c0f6510 2024-02-19T22:25:45.544 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.540+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70c107360 con 0x7fa70c0f6510 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.541+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6fc014420 con 0x7fa70c0f6510 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.541+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa70c045ad0 con 0x7fa70c0f6510 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.541+0000 7fa703fff700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.541+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa6fc040f40 con 0x7fa70c0f6510 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.541+0000 7fa70affd700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:45.545 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.542+0000 7fa70affd700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 0x7fa6f80323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa6f40096a0 tx=0x7fa6f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:45.548 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.546+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa6fc0095f0 con 0x7fa70c0f6510 2024-02-19T22:25:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.753+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa70c108210 con 0x7fa70c0f6510 2024-02-19T22:25:45.756 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.754+0000 7fa703fff700 1 -- 172.21.15.97:0/3887873564 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa6fc01db70 con 0x7fa70c0f6510 2024-02-19T22:25:45.756 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:45.756 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 msgr2=0x7fa6f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 0x7fa6f80323d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fa6f40096a0 tx=0x7fa6f4006b40).stop 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 msgr2=0x7fa70c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fa6fc005010 tx=0x7fa6fc00b040).stop 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 shutdown_connections 2024-02-19T22:25:45.758 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa6f802ff10 0x7fa6f80323d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 --2- 172.21.15.97:0/3887873564 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa70c0f6510 0x7fa70c1067e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 >> 172.21.15.97:0/3887873564 conn(0x7fa70c0f1c70 msgr2=0x7fa70c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 shutdown_connections 2024-02-19T22:25:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:45.756+0000 7fa712041700 1 -- 172.21.15.97:0/3887873564 wait complete. 2024-02-19T22:25:45.759 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:47.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:44.811567+0000 mgr.smithi045.bvlamo (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:47.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:46 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:45.754937+0000 mon.smithi045 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.97:0/3887873564' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:47.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:47.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:49.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:48 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:46.811951+0000 mgr.smithi045.bvlamo (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 -- 172.21.15.97:0/2614198720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 msgr2=0x7fd16c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 --2- 172.21.15.97:0/2614198720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c0f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7fd154009fb0 tx=0x7fd154009b30).stop 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 -- 172.21.15.97:0/2614198720 shutdown_connections 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 --2- 172.21.15.97:0/2614198720 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c0f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 -- 172.21.15.97:0/2614198720 >> 172.21.15.97:0/2614198720 conn(0x7fd16c0f0b70 msgr2=0x7fd16c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 -- 172.21.15.97:0/2614198720 shutdown_connections 2024-02-19T22:25:49.395 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.392+0000 7fd1714b9700 1 -- 172.21.15.97:0/2614198720 wait complete. 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.393+0000 7fd1714b9700 1 Processor -- start 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.393+0000 7fd1714b9700 1 -- start start 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.393+0000 7fd1714b9700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.393+0000 7fd1714b9700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c106d00 con 0x7fd16c0f5410 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd16affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd16affd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53544/0 (socket says 172.21.15.97:53544) 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd16affd700 1 -- 172.21.15.97:0/1184101510 learned_addr learned my addr 172.21.15.97:0/1184101510 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd16affd700 1 -- 172.21.15.97:0/1184101510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd154005040 con 0x7fd16c0f5410 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd16affd700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fd154005010 tx=0x7fd15400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:49.397 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd154009320 con 0x7fd16c0f5410 2024-02-19T22:25:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c106f00 con 0x7fd16c0f5410 2024-02-19T22:25:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd154013380 con 0x7fd16c0f5410 2024-02-19T22:25:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd154014e50 con 0x7fd16c0f5410 2024-02-19T22:25:49.398 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.394+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c107390 con 0x7fd16c0f5410 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.395+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd154014420 con 0x7fd16c0f5410 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.395+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16c045ad0 con 0x7fd16c0f5410 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.395+0000 7fd1637fe700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 0x7fd1580323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.395+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd154040e30 con 0x7fd16c0f5410 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.395+0000 7fd16a7fc700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 0x7fd1580323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:49.399 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.396+0000 7fd16a7fc700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 0x7fd1580323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd15c0096a0 tx=0x7fd15c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:49.402 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.400+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd1540095f0 con 0x7fd16c0f5410 2024-02-19T22:25:49.608 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.606+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd16c0fafa0 con 0x7fd16c0f5410 2024-02-19T22:25:49.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.607+0000 7fd1637fe700 1 -- 172.21.15.97:0/1184101510 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd15401db70 con 0x7fd16c0f5410 2024-02-19T22:25:49.609 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:49.609 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 msgr2=0x7fd1580323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 0x7fd1580323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fd15c0096a0 tx=0x7fd15c006b40).stop 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 msgr2=0x7fd16c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7fd154005010 tx=0x7fd15400b040).stop 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 shutdown_connections 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fd15802ff10 0x7fd1580323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 --2- 172.21.15.97:0/1184101510 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fd16c0f5410 0x7fd16c106810 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 >> 172.21.15.97:0/1184101510 conn(0x7fd16c0f0b70 msgr2=0x7fd16c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 shutdown_connections 2024-02-19T22:25:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:49.608+0000 7fd1714b9700 1 -- 172.21.15.97:0/1184101510 wait complete. 2024-02-19T22:25:49.612 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:50.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:49 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:49.607858+0000 mon.smithi045 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.97:0/1184101510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:51.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:50 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:48.812312+0000 mgr.smithi045.bvlamo (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:51.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:51.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:53.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:52 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:50.812697+0000 mgr.smithi045.bvlamo (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:53.364 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 -- 172.21.15.97:0/1193343020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 msgr2=0x7f70700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 --2- 172.21.15.97:0/1193343020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70700f5830 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f7060009fb0 tx=0x7f7060009b30).stop 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 -- 172.21.15.97:0/1193343020 shutdown_connections 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 --2- 172.21.15.97:0/1193343020 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70700f5830 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 -- 172.21.15.97:0/1193343020 >> 172.21.15.97:0/1193343020 conn(0x7f70700f0b70 msgr2=0x7f70700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 -- 172.21.15.97:0/1193343020 shutdown_connections 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.360+0000 7f7074c96700 1 -- 172.21.15.97:0/1193343020 wait complete. 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f7074c96700 1 Processor -- start 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f7074c96700 1 -- start start 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f7074c96700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f7074c96700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7070060240 con 0x7f70700f5410 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f706e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f706e59c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:53556/0 (socket says 172.21.15.97:53556) 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.361+0000 7f706e59c700 1 -- 172.21.15.97:0/630124947 learned_addr learned my addr 172.21.15.97:0/630124947 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:53.365 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f706e59c700 1 -- 172.21.15.97:0/630124947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7060005040 con 0x7f70700f5410 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f706e59c700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f7060005010 tx=0x7f706000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7060009320 con 0x7f70700f5410 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7070060440 con 0x7f70700f5410 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7060013380 con 0x7f70700f5410 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7060014e50 con 0x7f70700f5410 2024-02-19T22:25:53.366 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.362+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070109b90 con 0x7f70700f5410 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.363+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70600263a0 con 0x7f70700f5410 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.363+0000 7f7066ffd700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 0x7f705c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.363+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7070045ad0 con 0x7f70700f5410 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.363+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7060040ef0 con 0x7f70700f5410 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.363+0000 7f706dd9b700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 0x7f705c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:53.367 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.364+0000 7f706dd9b700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 0x7f705c0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f70580096a0 tx=0x7f7058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:53.370 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.368+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7060012920 con 0x7f70700f5410 2024-02-19T22:25:53.578 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.576+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f707010a3b0 con 0x7f70700f5410 2024-02-19T22:25:53.579 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.576+0000 7f7066ffd700 1 -- 172.21.15.97:0/630124947 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7060012330 con 0x7f70700f5410 2024-02-19T22:25:53.579 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:53.579 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 msgr2=0x7f705c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 0x7f705c0323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f70580096a0 tx=0x7f7058006b40).stop 2024-02-19T22:25:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 msgr2=0x7f70701094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f7060005010 tx=0x7f706000b040).stop 2024-02-19T22:25:53.581 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 shutdown_connections 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f705c02ff10 0x7f705c0323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 --2- 172.21.15.97:0/630124947 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f70700f5410 0x7f70701094e0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 >> 172.21.15.97:0/630124947 conn(0x7f70700f0b70 msgr2=0x7f70700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 shutdown_connections 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:53.578+0000 7f7074c96700 1 -- 172.21.15.97:0/630124947 wait complete. 2024-02-19T22:25:53.582 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:54.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:53 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:53.577603+0000 mon.smithi045 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.97:0/630124947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:55.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:54 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:52.813083+0000 mgr.smithi045.bvlamo (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:55.289 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:55.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:56.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:55 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:54.813379+0000 mgr.smithi045.bvlamo (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:57.179 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 -- 172.21.15.97:0/9244347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 msgr2=0x7f4f480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/9244347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f480f6930 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f4f3c009fb0 tx=0x7f4f3c009b30).stop 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 -- 172.21.15.97:0/9244347 shutdown_connections 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/9244347 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f480f6930 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 -- 172.21.15.97:0/9244347 >> 172.21.15.97:0/9244347 conn(0x7f4f480f1c70 msgr2=0x7f4f480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 -- 172.21.15.97:0/9244347 shutdown_connections 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.165+0000 7f4f4d0af700 1 -- 172.21.15.97:0/9244347 wait complete. 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f4d0af700 1 Processor -- start 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f4d0af700 1 -- start start 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f4d0af700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f4d0af700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f48069e20 con 0x7f4f480f6510 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f46d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:57.180 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f46d9d700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46836/0 (socket says 172.21.15.97:46836) 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.166+0000 7f4f46d9d700 1 -- 172.21.15.97:0/2228685097 learned_addr learned my addr 172.21.15.97:0/2228685097 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f46d9d700 1 -- 172.21.15.97:0/2228685097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c005040 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f46d9d700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f4f3c005010 tx=0x7f4f3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c009320 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f48069fc0 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f3c013380 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c014e50 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.167+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4806a450 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f48059100 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f3c0095f0 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f2f7fe700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 0x7f4f300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f3c040e90 con 0x7f4f480f6510 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f4659c700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 0x7f4f300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:57.181 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.168+0000 7f4f4659c700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 0x7f4f300323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:57.182 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.172+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f3c014420 con 0x7f4f480f6510 2024-02-19T22:25:57.381 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.379+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f480fc0a0 con 0x7f4f480f6510 2024-02-19T22:25:57.382 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.379+0000 7f4f2f7fe700 1 -- 172.21.15.97:0/2228685097 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4f3c010230 con 0x7f4f480f6510 2024-02-19T22:25:57.382 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:25:57.382 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 msgr2=0x7f4f300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 0x7f4f300323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).stop 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 msgr2=0x7f4f4806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f4f3c005010 tx=0x7f4f3c00b040).stop 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 shutdown_connections 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4f3002ff10 0x7f4f300323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 --2- 172.21.15.97:0/2228685097 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4f480f6510 0x7f4f4806b9a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 >> 172.21.15.97:0/2228685097 conn(0x7f4f480f1c70 msgr2=0x7f4f480f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:57.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 shutdown_connections 2024-02-19T22:25:57.384 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:25:57.381+0000 7f4f4d0af700 1 -- 172.21.15.97:0/2228685097 wait complete. 2024-02-19T22:25:57.384 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:25:58.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:56.813713+0000 mgr.smithi045.bvlamo (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:58.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:57 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:25:57.380312+0000 mon.smithi045 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.97:0/2228685097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:59.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:59.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:00.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:25:59 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:25:58.814090+0000 mgr.smithi045.bvlamo (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 -- 172.21.15.97:0/1980485596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 msgr2=0x7fa1240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 --2- 172.21.15.97:0/1980485596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1240f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa114009fb0 tx=0x7fa114009b30).stop 2024-02-19T22:26:01.191 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 -- 172.21.15.97:0/1980485596 shutdown_connections 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 --2- 172.21.15.97:0/1980485596 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1240f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 -- 172.21.15.97:0/1980485596 >> 172.21.15.97:0/1980485596 conn(0x7fa1240f0b70 msgr2=0x7fa1240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 -- 172.21.15.97:0/1980485596 shutdown_connections 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.188+0000 7fa128a21700 1 -- 172.21.15.97:0/1980485596 wait complete. 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa128a21700 1 Processor -- start 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa128a21700 1 -- start start 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa128a21700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa128a21700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa124105bd0 con 0x7fa1240f5410 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa12259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa12259c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46844/0 (socket says 172.21.15.97:46844) 2024-02-19T22:26:01.192 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.189+0000 7fa12259c700 1 -- 172.21.15.97:0/2865968285 learned_addr learned my addr 172.21.15.97:0/2865968285 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa12259c700 1 -- 172.21.15.97:0/2865968285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114005040 con 0x7fa1240f5410 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa12259c700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa114005010 tx=0x7fa11400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114009320 con 0x7fa1240f5410 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa124105dd0 con 0x7fa1240f5410 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa114013380 con 0x7fa1240f5410 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa114014e50 con 0x7fa1240f5410 2024-02-19T22:26:01.194 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.190+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa124106260 con 0x7fa1240f5410 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1140263a0 con 0x7fa1240f5410 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa11affd700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa124045ad0 con 0x7fa1240f5410 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa114041270 con 0x7fa1240f5410 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa121d9b700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:01.195 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.191+0000 7fa121d9b700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 0x7fa1100323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa10c0096a0 tx=0x7fa10c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:01.198 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.196+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa11401cb10 con 0x7fa1240f5410 2024-02-19T22:26:01.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.401+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa124107110 con 0x7fa1240f5410 2024-02-19T22:26:01.404 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.402+0000 7fa11affd700 1 -- 172.21.15.97:0/2865968285 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa11401cb10 con 0x7fa1240f5410 2024-02-19T22:26:01.405 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:01.405 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 msgr2=0x7fa1100323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 0x7fa1100323d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa10c0096a0 tx=0x7fa10c006b40).stop 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 msgr2=0x7fa1241056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa114005010 tx=0x7fa11400b040).stop 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 shutdown_connections 2024-02-19T22:26:01.406 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fa11002ff10 0x7fa1100323d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 --2- 172.21.15.97:0/2865968285 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fa1240f5410 0x7fa1241056e0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 >> 172.21.15.97:0/2865968285 conn(0x7fa1240f0b70 msgr2=0x7fa1240f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 shutdown_connections 2024-02-19T22:26:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:01.404+0000 7fa128a21700 1 -- 172.21.15.97:0/2865968285 wait complete. 2024-02-19T22:26:01.407 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:02.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:00.814483+0000 mgr.smithi045.bvlamo (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:02.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:01 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:01.403409+0000 mon.smithi045 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.97:0/2865968285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:03.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:03.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:02.814750+0000 mgr.smithi045.bvlamo (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:03.942249+0000 mon.smithi045 (mon.0) 331 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:26:04.891 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:04 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:03.942492+0000 mon.smithi045 (mon.0) 332 : audit [INF] from='mgr.14162 172.21.15.45:0/2536902737' entity='mgr.smithi045.bvlamo' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi045.bvlamo/trash_purge_schedule"}]: dispatch 2024-02-19T22:26:04.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 -- 172.21.15.97:0/378783492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 msgr2=0x7f0e3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:04.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 --2- 172.21.15.97:0/378783492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c0f5830 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f0e34009fb0 tx=0x7f0e34009b30).stop 2024-02-19T22:26:04.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 -- 172.21.15.97:0/378783492 shutdown_connections 2024-02-19T22:26:04.996 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 --2- 172.21.15.97:0/378783492 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c0f5830 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 -- 172.21.15.97:0/378783492 >> 172.21.15.97:0/378783492 conn(0x7f0e3c0f0b70 msgr2=0x7f0e3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 -- 172.21.15.97:0/378783492 shutdown_connections 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.993+0000 7f0e438b6700 1 -- 172.21.15.97:0/378783492 wait complete. 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e438b6700 1 Processor -- start 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e438b6700 1 -- start start 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e438b6700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e438b6700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c106cc0 con 0x7f0e3c0f5410 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e41652700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e41652700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:46864/0 (socket says 172.21.15.97:46864) 2024-02-19T22:26:04.997 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e41652700 1 -- 172.21.15.97:0/325200273 learned_addr learned my addr 172.21.15.97:0/325200273 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:04.998 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e41652700 1 -- 172.21.15.97:0/325200273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e34005040 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.994+0000 7f0e41652700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0e34005010 tx=0x7f0e3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.995+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34009320 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.995+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c106ec0 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.995+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e34013380 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.995+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e34014e50 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.995+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c107350 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c045ad0 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0e340095f0 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e39ffb700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 0x7f0e240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0e34040fa0 con 0x7f0e3c0f5410 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e40e51700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 0x7f0e240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:04.999 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:04.996+0000 7f0e40e51700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 0x7f0e240323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0e300096a0 tx=0x7f0e30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:05.003 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.001+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0e34014420 con 0x7f0e3c0f5410 2024-02-19T22:26:05.208 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.206+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e3c108200 con 0x7f0e3c0f5410 2024-02-19T22:26:05.209 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.206+0000 7f0e39ffb700 1 -- 172.21.15.97:0/325200273 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e34010230 con 0x7f0e3c0f5410 2024-02-19T22:26:05.209 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:05.209 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 msgr2=0x7f0e240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 0x7f0e240323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f0e300096a0 tx=0x7f0e30006b40).stop 2024-02-19T22:26:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 msgr2=0x7f0e3c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:05.210 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f0e34005010 tx=0x7f0e3400b040).stop 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 shutdown_connections 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f0e2402ff10 0x7f0e240323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 --2- 172.21.15.97:0/325200273 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f0e3c0f5410 0x7f0e3c1067d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 >> 172.21.15.97:0/325200273 conn(0x7f0e3c0f0b70 msgr2=0x7f0e3c05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 shutdown_connections 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:05.208+0000 7f0e438b6700 1 -- 172.21.15.97:0/325200273 wait complete. 2024-02-19T22:26:05.211 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:06.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:05 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:05.207682+0000 mon.smithi045 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.97:0/325200273' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:06.824 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:06.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:07.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:06 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:04.815097+0000 mgr.smithi045.bvlamo (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.789+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1451772052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 msgr2=0x7f4cbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.789+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1451772052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc0f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f4cac009fb0 tx=0x7f4cac009b30).stop 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.789+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1451772052 shutdown_connections 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.789+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1451772052 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc0f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1451772052 >> 172.21.15.97:0/1451772052 conn(0x7f4cbc0f1c70 msgr2=0x7f4cbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1451772052 shutdown_connections 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1451772052 wait complete. 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 Processor -- start 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 -- start start 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:08.809 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc46fc700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc106cd0 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc2498700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc2498700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42258/0 (socket says 172.21.15.97:42258) 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.790+0000 7f4cc2498700 1 -- 172.21.15.97:0/1227442819 learned_addr learned my addr 172.21.15.97:0/1227442819 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cc2498700 1 -- 172.21.15.97:0/1227442819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cac005040 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cc2498700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f4cac005010 tx=0x7f4cac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac009320 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cac013380 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc106ed0 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cac014e50 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.791+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc107360 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.792+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc045ad0 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.792+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4cac0095f0 con 0x7f4cbc0f6510 2024-02-19T22:26:08.810 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.792+0000 7f4cb2ffd700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 0x7f4ca80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.792+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4cac040e90 con 0x7f4cbc0f6510 2024-02-19T22:26:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.792+0000 7f4cc1c97700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 0x7f4ca80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.793+0000 7f4cc1c97700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 0x7f4ca80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4cb80096a0 tx=0x7f4cb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:08.811 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:08.797+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4cac014420 con 0x7f4cbc0f6510 2024-02-19T22:26:09.005 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.003+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4cbc108210 con 0x7f4cbc0f6510 2024-02-19T22:26:09.006 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.003+0000 7f4cb2ffd700 1 -- 172.21.15.97:0/1227442819 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4cac010230 con 0x7f4cbc0f6510 2024-02-19T22:26:09.006 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:09.006 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:09.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 msgr2=0x7f4ca80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:09.007 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 0x7f4ca80323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4cb80096a0 tx=0x7f4cb8006b40).stop 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 msgr2=0x7f4cbc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f4cac005010 tx=0x7f4cac00b040).stop 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 shutdown_connections 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f4ca802ff10 0x7f4ca80323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 --2- 172.21.15.97:0/1227442819 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f4cbc0f6510 0x7f4cbc1067e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 >> 172.21.15.97:0/1227442819 conn(0x7f4cbc0f1c70 msgr2=0x7f4cbc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 shutdown_connections 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:09.005+0000 7f4cc46fc700 1 -- 172.21.15.97:0/1227442819 wait complete. 2024-02-19T22:26:09.008 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:09.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:08 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:06.815477+0000 mgr.smithi045.bvlamo (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:10.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:09 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:09.004561+0000 mon.smithi045 (mon.0) 334 : audit [DBG] from='client.? 172.21.15.97:0/1227442819' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:10.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:10.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:11.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:10 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:08.815778+0000 mgr.smithi045.bvlamo (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 -- 172.21.15.97:0/1407794778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c0f5410 msgr2=0x7f780c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 --2- 172.21.15.97:0/1407794778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c0f5410 0x7f780c0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f77fc009960 tx=0x7f77fc004d10).stop 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 -- 172.21.15.97:0/1407794778 shutdown_connections 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 --2- 172.21.15.97:0/1407794778 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c0f5410 0x7f780c0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 -- 172.21.15.97:0/1407794778 >> 172.21.15.97:0/1407794778 conn(0x7f780c0f0b70 msgr2=0x7f780c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 -- 172.21.15.97:0/1407794778 shutdown_connections 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.621+0000 7f7811c2a700 1 -- 172.21.15.97:0/1407794778 wait complete. 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f7811c2a700 1 Processor -- start 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f7811c2a700 1 -- start start 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f7811c2a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:12.625 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f7811c2a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c109af0 con 0x7f780c060240 2024-02-19T22:26:12.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f780b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:12.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f780b7fe700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:42270/0 (socket says 172.21.15.97:42270) 2024-02-19T22:26:12.626 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.622+0000 7f780b7fe700 1 -- 172.21.15.97:0/3228777599 learned_addr learned my addr 172.21.15.97:0/3228777599 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f780b7fe700 1 -- 172.21.15.97:0/3228777599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77fc009640 con 0x7f780c060240 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f780b7fe700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f77fc009930 tx=0x7f77fc004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77fc019070 con 0x7f780c060240 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77fc007e90 con 0x7f780c060240 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c109c30 con 0x7f780c060240 2024-02-19T22:26:12.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77fc01e420 con 0x7f780c060240 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.623+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c10a0c0 con 0x7f780c060240 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.624+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77fc025060 con 0x7f780c060240 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.624+0000 7f7803fff700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 0x7f77f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.624+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c045ad0 con 0x7f780c060240 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.624+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77fc041310 con 0x7f780c060240 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.624+0000 7f780affd700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 0x7f77f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:12.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.625+0000 7f780affd700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 0x7f77f8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f77f40096a0 tx=0x7f77f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:12.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.629+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f77fc01e580 con 0x7f780c060240 2024-02-19T22:26:12.837 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.835+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f780c0fafa0 con 0x7f780c060240 2024-02-19T22:26:12.838 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.835+0000 7f7803fff700 1 -- 172.21.15.97:0/3228777599 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f77fc011b10 con 0x7f780c060240 2024-02-19T22:26:12.838 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:12.838 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 msgr2=0x7f77f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 0x7f77f8032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f77f40096a0 tx=0x7f77f4006b40).stop 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 msgr2=0x7f780c109600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f77fc009930 tx=0x7f77fc004d10).stop 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 shutdown_connections 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f77f802ff60 0x7f77f8032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 --2- 172.21.15.97:0/3228777599 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f780c060240 0x7f780c109600 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 >> 172.21.15.97:0/3228777599 conn(0x7f780c0f0b70 msgr2=0x7f780c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 shutdown_connections 2024-02-19T22:26:12.840 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:12.837+0000 7f7811c2a700 1 -- 172.21.15.97:0/3228777599 wait complete. 2024-02-19T22:26:12.841 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:13.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:12 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:10.816152+0000 mgr.smithi045.bvlamo (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:14.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:13 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:12.836677+0000 mon.smithi045 (mon.0) 335 : audit [DBG] from='client.? 172.21.15.97:0/3228777599' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:14.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:14.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:15.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:14 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:12.816570+0000 mgr.smithi045.bvlamo (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 -- 172.21.15.97:0/1024584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80f6510 msgr2=0x7f80f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/1024584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80f6510 0x7f80f80f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f80e8009fb0 tx=0x7f80e8009b30).stop 2024-02-19T22:26:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 -- 172.21.15.97:0/1024584663 shutdown_connections 2024-02-19T22:26:16.603 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/1024584663 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f80f6510 0x7f80f80f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 -- 172.21.15.97:0/1024584663 >> 172.21.15.97:0/1024584663 conn(0x7f80f80f1c70 msgr2=0x7f80f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 -- 172.21.15.97:0/1024584663 shutdown_connections 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.599+0000 7f80fe3aa700 1 -- 172.21.15.97:0/1024584663 wait complete. 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80fe3aa700 1 Processor -- start 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80fe3aa700 1 -- start start 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80fe3aa700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80fe3aa700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f806bd80 con 0x7f80f806b960 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80f7fff700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:59368/0 (socket says 172.21.15.97:59368) 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.600+0000 7f80f7fff700 1 -- 172.21.15.97:0/290744759 learned_addr learned my addr 172.21.15.97:0/290744759 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80f7fff700 1 -- 172.21.15.97:0/290744759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e8005040 con 0x7f80f806b960 2024-02-19T22:26:16.604 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80f7fff700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f80e8009f80 tx=0x7f80e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:16.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80e8013380 con 0x7f80f806b960 2024-02-19T22:26:16.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f806a310 con 0x7f80f806b960 2024-02-19T22:26:16.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80e8013790 con 0x7f80f806b960 2024-02-19T22:26:16.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80e8014420 con 0x7f80f806b960 2024-02-19T22:26:16.605 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.601+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f806a800 con 0x7f80f806b960 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80e8025060 con 0x7f80f806b960 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f8045ad0 con 0x7f80f806b960 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80f4ff9700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 0x7f80e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80e8040d70 con 0x7f80f806b960 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80f77fe700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 0x7f80e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:16.606 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.602+0000 7f80f77fe700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 0x7f80e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f80e00096a0 tx=0x7f80e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:16.609 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.607+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80e801cb60 con 0x7f80f806b960 2024-02-19T22:26:16.815 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.813+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80f810cae0 con 0x7f80f806b960 2024-02-19T22:26:16.816 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.813+0000 7f80f4ff9700 1 -- 172.21.15.97:0/290744759 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f80e802d2d0 con 0x7f80f806b960 2024-02-19T22:26:16.816 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:16.816 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:16.817 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 msgr2=0x7f80e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 0x7f80e4032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f80e00096a0 tx=0x7f80e0006b40).stop 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 msgr2=0x7f80f8069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f80e8009f80 tx=0x7f80e8009b30).stop 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 shutdown_connections 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f80e402ff60 0x7f80e4032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 --2- 172.21.15.97:0/290744759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f80f806b960 0x7f80f8069e20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 >> 172.21.15.97:0/290744759 conn(0x7f80f80f1c70 msgr2=0x7f80f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 shutdown_connections 2024-02-19T22:26:16.818 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:16.815+0000 7f80fe3aa700 1 -- 172.21.15.97:0/290744759 wait complete. 2024-02-19T22:26:16.819 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:17.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:16 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:14.817012+0000 mgr.smithi045.bvlamo (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:18.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:17 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:16.814652+0000 mon.smithi045 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.97:0/290744759' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:18.626 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:18.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:19.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:18 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:16.817269+0000 mgr.smithi045.bvlamo (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:20.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 -- 172.21.15.97:0/1137244759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 msgr2=0x7f71540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:20.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 --2- 172.21.15.97:0/1137244759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f71540f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f7148009fb0 tx=0x7f7148009b30).stop 2024-02-19T22:26:20.561 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 -- 172.21.15.97:0/1137244759 shutdown_connections 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 --2- 172.21.15.97:0/1137244759 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f71540f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 -- 172.21.15.97:0/1137244759 >> 172.21.15.97:0/1137244759 conn(0x7f71540f0b70 msgr2=0x7f71540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 -- 172.21.15.97:0/1137244759 shutdown_connections 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.557+0000 7f715b93e700 1 -- 172.21.15.97:0/1137244759 wait complete. 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.558+0000 7f715b93e700 1 Processor -- start 2024-02-19T22:26:20.562 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.558+0000 7f715b93e700 1 -- start start 2024-02-19T22:26:20.563 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.558+0000 7f715b93e700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.558+0000 7f715b93e700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154060220 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.558+0000 7f71596da700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f71596da700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:59388/0 (socket says 172.21.15.97:59388) 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f71596da700 1 -- 172.21.15.97:0/2158598995 learned_addr learned my addr 172.21.15.97:0/2158598995 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f71596da700 1 -- 172.21.15.97:0/2158598995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148005040 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f71596da700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7148005010 tx=0x7f714800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148009320 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154060420 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7148013380 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7148014e50 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.559+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7154109bb0 con 0x7f71540f5410 2024-02-19T22:26:20.564 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.560+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71480095f0 con 0x7f71540f5410 2024-02-19T22:26:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.560+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7154059100 con 0x7f71540f5410 2024-02-19T22:26:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.560+0000 7f7145ffb700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 0x7f71400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.560+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7148040fa0 con 0x7f71540f5410 2024-02-19T22:26:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.560+0000 7f7158ed9700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 0x7f71400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:20.565 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.561+0000 7f7158ed9700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 0x7f71400323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f71500096a0 tx=0x7f7150006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:20.568 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.566+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7148014420 con 0x7f71540f5410 2024-02-19T22:26:20.773 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.770+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7154045ad0 con 0x7f71540f5410 2024-02-19T22:26:20.774 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.770+0000 7f7145ffb700 1 -- 172.21.15.97:0/2158598995 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7148010230 con 0x7f71540f5410 2024-02-19T22:26:20.774 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:20.774 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:20.775 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 msgr2=0x7f71400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 0x7f71400323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f71500096a0 tx=0x7f7150006b40).stop 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 msgr2=0x7f7154109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f7148005010 tx=0x7f714800b040).stop 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 shutdown_connections 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f714002ff10 0x7f71400323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 --2- 172.21.15.97:0/2158598995 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f71540f5410 0x7f7154109500 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 >> 172.21.15.97:0/2158598995 conn(0x7f71540f0b70 msgr2=0x7f71540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 shutdown_connections 2024-02-19T22:26:20.776 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:20.772+0000 7f715b93e700 1 -- 172.21.15.97:0/2158598995 wait complete. 2024-02-19T22:26:20.777 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:21.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:20 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:18.817552+0000 mgr.smithi045.bvlamo (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:22.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:21 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:20.772636+0000 mon.smithi045 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.97:0/2158598995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:22.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:22.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:23.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:22 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:20.817835+0000 mgr.smithi045.bvlamo (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 -- 172.21.15.97:0/218543967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 msgr2=0x7f9e540f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 --2- 172.21.15.97:0/218543967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e540f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f9e44009fb0 tx=0x7f9e44009b30).stop 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 -- 172.21.15.97:0/218543967 shutdown_connections 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 --2- 172.21.15.97:0/218543967 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e540f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 -- 172.21.15.97:0/218543967 >> 172.21.15.97:0/218543967 conn(0x7f9e540f1c70 msgr2=0x7f9e540f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 -- 172.21.15.97:0/218543967 shutdown_connections 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.527+0000 7f9e5c593700 1 -- 172.21.15.97:0/218543967 wait complete. 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5c593700 1 Processor -- start 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5c593700 1 -- start start 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5c593700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:24.555 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5c593700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e54069e20 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5a32f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5a32f700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:59392/0 (socket says 172.21.15.97:59392) 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5a32f700 1 -- 172.21.15.97:0/712964028 learned_addr learned my addr 172.21.15.97:0/712964028 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5a32f700 1 -- 172.21.15.97:0/712964028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e44005040 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e5a32f700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9e44005010 tx=0x7f9e4400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.528+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e44009320 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9e44013380 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e54069fc0 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e44014e50 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5406a450 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e440095f0 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.529+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e54059100 con 0x7f9e540f6510 2024-02-19T22:26:24.556 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.530+0000 7f9e4affd700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 0x7f9e400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.530+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e44040fa0 con 0x7f9e540f6510 2024-02-19T22:26:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.530+0000 7f9e59b2e700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 0x7f9e400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.530+0000 7f9e59b2e700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 0x7f9e400323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:24.557 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.534+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e44014420 con 0x7f9e540f6510 2024-02-19T22:26:24.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.737+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e540fc0a0 con 0x7f9e540f6510 2024-02-19T22:26:24.740 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.738+0000 7f9e4affd700 1 -- 172.21.15.97:0/712964028 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e44010230 con 0x7f9e540f6510 2024-02-19T22:26:24.741 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:24.741 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 msgr2=0x7f9e400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 0x7f9e400323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f9e500096a0 tx=0x7f9e50006b40).stop 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 msgr2=0x7f9e5406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f9e44005010 tx=0x7f9e4400b040).stop 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 shutdown_connections 2024-02-19T22:26:24.742 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f9e4002ff10 0x7f9e400323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 --2- 172.21.15.97:0/712964028 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f9e540f6510 0x7f9e5406b9a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 >> 172.21.15.97:0/712964028 conn(0x7f9e540f1c70 msgr2=0x7f9e540f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 shutdown_connections 2024-02-19T22:26:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:24.740+0000 7f9e5c593700 1 -- 172.21.15.97:0/712964028 wait complete. 2024-02-19T22:26:24.743 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:25.058 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:24 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:22.818083+0000 mgr.smithi045.bvlamo (mgr.14162) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:26.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:25 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:24.739481+0000 mon.smithi045 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.97:0/712964028' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:26.463 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:26.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:27.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:26 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:24.818443+0000 mgr.smithi045.bvlamo (mgr.14162) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:28.480 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 -- 172.21.15.97:0/3461863220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 msgr2=0x7fbaf80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 --2- 172.21.15.97:0/3461863220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf80f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fbae8009fb0 tx=0x7fbae8009b30).stop 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 -- 172.21.15.97:0/3461863220 shutdown_connections 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 --2- 172.21.15.97:0/3461863220 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf80f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 -- 172.21.15.97:0/3461863220 >> 172.21.15.97:0/3461863220 conn(0x7fbaf80f0b70 msgr2=0x7fbaf80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 -- 172.21.15.97:0/3461863220 shutdown_connections 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.477+0000 7fbaffee7700 1 -- 172.21.15.97:0/3461863220 wait complete. 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbaffee7700 1 Processor -- start 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbaffee7700 1 -- start start 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbaffee7700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbaffee7700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf8106cd0 con 0x7fbaf80f5410 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbafdc83700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbafdc83700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:47358/0 (socket says 172.21.15.97:47358) 2024-02-19T22:26:28.481 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.478+0000 7fbafdc83700 1 -- 172.21.15.97:0/1023102511 learned_addr learned my addr 172.21.15.97:0/1023102511 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbafdc83700 1 -- 172.21.15.97:0/1023102511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae8005040 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbafdc83700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fbae8005010 tx=0x7fbae800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae8009320 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf8106ed0 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbae8013380 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbae8014e50 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.479+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf8107360 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.480+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf8045ad0 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.480+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbae80095f0 con 0x7fbaf80f5410 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.480+0000 7fbaee7fc700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 0x7fbae40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:28.483 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.480+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbae8040e90 con 0x7fbaf80f5410 2024-02-19T22:26:28.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.480+0000 7fbafd482700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 0x7fbae40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:28.484 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.481+0000 7fbafd482700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 0x7fbae40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fbaf40096a0 tx=0x7fbaf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:28.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.485+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbae8014420 con 0x7fbaf80f5410 2024-02-19T22:26:28.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.689+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbaf80fafa0 con 0x7fbaf80f5410 2024-02-19T22:26:28.692 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.690+0000 7fbaee7fc700 1 -- 172.21.15.97:0/1023102511 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbae8010230 con 0x7fbaf80f5410 2024-02-19T22:26:28.693 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:28.693 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 msgr2=0x7fbae40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 0x7fbae40323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fbaf40096a0 tx=0x7fbaf4006b40).stop 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 msgr2=0x7fbaf81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fbae8005010 tx=0x7fbae800b040).stop 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 shutdown_connections 2024-02-19T22:26:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fbae402ff10 0x7fbae40323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 --2- 172.21.15.97:0/1023102511 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fbaf80f5410 0x7fbaf81067e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 >> 172.21.15.97:0/1023102511 conn(0x7fbaf80f0b70 msgr2=0x7fbaf805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 shutdown_connections 2024-02-19T22:26:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:28.692+0000 7fbaffee7700 1 -- 172.21.15.97:0/1023102511 wait complete. 2024-02-19T22:26:28.695 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:29.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:28 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:26.818761+0000 mgr.smithi045.bvlamo (mgr.14162) 382 : cluster [DBG] pgmap v366: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:30.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:29 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:28.691235+0000 mon.smithi045 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.97:0/1023102511' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:30.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:30.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:31.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:30 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:28.819054+0000 mgr.smithi045.bvlamo (mgr.14162) 383 : cluster [DBG] pgmap v367: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 -- 172.21.15.97:0/3573633922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 msgr2=0x7f94a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 --2- 172.21.15.97:0/3573633922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a80f6930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f9498009fb0 tx=0x7f9498009b30).stop 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 -- 172.21.15.97:0/3573633922 shutdown_connections 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 --2- 172.21.15.97:0/3573633922 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a80f6930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 -- 172.21.15.97:0/3573633922 >> 172.21.15.97:0/3573633922 conn(0x7f94a80f1c70 msgr2=0x7f94a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:32.487 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 -- 172.21.15.97:0/3573633922 shutdown_connections 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.469+0000 7f94ad61c700 1 -- 172.21.15.97:0/3573633922 wait complete. 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94ad61c700 1 Processor -- start 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94ad61c700 1 -- start start 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94ad61c700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94ad61c700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a8069e20 con 0x7f94a80f6510 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94a6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94a6ffd700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:47376/0 (socket says 172.21.15.97:47376) 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.470+0000 7f94a6ffd700 1 -- 172.21.15.97:0/505348477 learned_addr learned my addr 172.21.15.97:0/505348477 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f94a6ffd700 1 -- 172.21.15.97:0/505348477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9498005040 con 0x7f94a80f6510 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f94a6ffd700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9498005010 tx=0x7f949800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9498009320 con 0x7f94a80f6510 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a8069fc0 con 0x7f94a80f6510 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9498013380 con 0x7f94a80f6510 2024-02-19T22:26:32.488 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9498014e50 con 0x7f94a80f6510 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.471+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a806a450 con 0x7f94a80f6510 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.472+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f94980095f0 con 0x7f94a80f6510 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.472+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a8059100 con 0x7f94a80f6510 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.472+0000 7f949f7fe700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 0x7f94940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.472+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9498040fa0 con 0x7f94a80f6510 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.472+0000 7f94a67fc700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 0x7f94940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.473+0000 7f94a67fc700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 0x7f94940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f94900096a0 tx=0x7f9490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:32.489 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.476+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9498014420 con 0x7f94a80f6510 2024-02-19T22:26:32.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.681+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94a80fc0a0 con 0x7f94a80f6510 2024-02-19T22:26:32.684 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.682+0000 7f949f7fe700 1 -- 172.21.15.97:0/505348477 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9498010230 con 0x7f94a80f6510 2024-02-19T22:26:32.685 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:32.685 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 msgr2=0x7f94940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 0x7f94940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f94900096a0 tx=0x7f9490006b40).stop 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 msgr2=0x7f94a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f9498005010 tx=0x7f949800b040).stop 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 shutdown_connections 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7f949402ff10 0x7f94940323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 --2- 172.21.15.97:0/505348477 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7f94a80f6510 0x7f94a806b9a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 >> 172.21.15.97:0/505348477 conn(0x7f94a80f1c70 msgr2=0x7f94a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.684+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 shutdown_connections 2024-02-19T22:26:32.687 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:32.685+0000 7f94ad61c700 1 -- 172.21.15.97:0/505348477 wait complete. 2024-02-19T22:26:32.688 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:33.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:32 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:30.819322+0000 mgr.smithi045.bvlamo (mgr.14162) 384 : cluster [DBG] pgmap v368: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:34.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:33 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:32.683223+0000 mon.smithi045 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.97:0/505348477' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:34.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:26:34.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6020aec-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:26:35.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:34 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:32.819713+0000 mgr.smithi045.bvlamo (mgr.14162) 385 : cluster [DBG] pgmap v369: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:36.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- 172.21.15.97:0/2703446783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 msgr2=0x7fae480f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:36.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 --2- 172.21.15.97:0/2703446783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae480f9190 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fae44009fb0 tx=0x7fae44009b30).stop 2024-02-19T22:26:36.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- 172.21.15.97:0/2703446783 shutdown_connections 2024-02-19T22:26:36.414 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 --2- 172.21.15.97:0/2703446783 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae480f9190 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- 172.21.15.97:0/2703446783 >> 172.21.15.97:0/2703446783 conn(0x7fae480f0b70 msgr2=0x7fae480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- 172.21.15.97:0/2703446783 shutdown_connections 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- 172.21.15.97:0/2703446783 wait complete. 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 Processor -- start 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.409+0000 7fae5026a700 1 -- start start 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae5026a700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae5026a700 1 -- --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae48106cc0 con 0x7fae480601e0 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae4e006700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae4e006700 1 --2- >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.45:3300/0 says I am v2:172.21.15.97:39906/0 (socket says 172.21.15.97:39906) 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae4e006700 1 -- 172.21.15.97:0/2039653888 learned_addr learned my addr 172.21.15.97:0/2039653888 (peer_addr_for_me v2:172.21.15.97:0/0) 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae4e006700 1 -- 172.21.15.97:0/2039653888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae44005040 con 0x7fae480601e0 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.410+0000 7fae4e006700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fae44007a10 tx=0x7fae44009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae44018070 con 0x7fae480601e0 2024-02-19T22:26:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae44013410 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae48106ec0 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4401c3c0 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae48107350 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.411+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae44013ca0 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.412+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae48045ad0 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.412+0000 7fae3effd700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 0x7fae34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.412+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae440414b0 con 0x7fae480601e0 2024-02-19T22:26:36.416 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.412+0000 7fae4d805700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 0x7fae34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:26:36.417 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.412+0000 7fae4d805700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 0x7fae34032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fae380096a0 tx=0x7fae38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:26:36.419 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.417+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae4401d420 con 0x7fae480601e0 2024-02-19T22:26:36.627 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.625+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 --> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae481081a0 con 0x7fae480601e0 2024-02-19T22:26:36.628 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.625+0000 7fae3effd700 1 -- 172.21.15.97:0/2039653888 <== mon.0 v2:172.21.15.45:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fae44025ce0 con 0x7fae480601e0 2024-02-19T22:26:36.628 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-02-19T22:26:36.629 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"f6020aec-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:54.148702Z","created":"2024-02-19T22:12:54.148702Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi045","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.45:3300","nonce":0},{"type":"v1","addr":"172.21.15.45:6789","nonce":0}]},"addr":"172.21.15.45:6789/0","public_addr":"172.21.15.45:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:26:36.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 msgr2=0x7fae34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:36.630 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 0x7fae34032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fae380096a0 tx=0x7fae38006b40).stop 2024-02-19T22:26:36.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 msgr2=0x7fae481067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:26:36.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fae44007a10 tx=0x7fae44009b30).stop 2024-02-19T22:26:36.631 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 shutdown_connections 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:6800/3887107005,v1:172.21.15.45:6801/3887107005] conn(0x7fae3402ff60 0x7fae34032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 --2- 172.21.15.97:0/2039653888 >> [v2:172.21.15.45:3300/0,v1:172.21.15.45:6789/0] conn(0x7fae480601e0 0x7fae481067d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 >> 172.21.15.97:0/2039653888 conn(0x7fae480f0b70 msgr2=0x7fae480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 shutdown_connections 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:2024-02-19T22:26:36.627+0000 7fae5026a700 1 -- 172.21.15.97:0/2039653888 wait complete. 2024-02-19T22:26:36.632 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-02-19T22:26:37.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: cluster 2024-02-19T22:26:34.820023+0000 mgr.smithi045.bvlamo (mgr.14162) 386 : cluster [DBG] pgmap v370: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:26:37.141 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:36 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: audit 2024-02-19T22:26:36.626396+0000 mon.smithi045 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.97:0/2039653888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:26:37.253 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:26:37.254 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:26:37.254 DEBUG:teuthology.orchestra.run.smithi045:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:26:37.279 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:26:37.298 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:26:37.298 INFO:tasks.cephadm.mon.smithi045:Stopping mon.smithi045... 2024-02-19T22:26:37.298 DEBUG:teuthology.orchestra.run.smithi045:> sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045 2024-02-19T22:26:37.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:37 smithi045 systemd[1]: Stopping Ceph mon.smithi045 for f6020aec-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:26:37.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:26:37.486+0000 7fda86277700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi045 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:26:37.641 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:37 smithi045 ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045[38092]: debug 2024-02-19T22:26:37.486+0000 7fda86277700 -1 mon.smithi045@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:26:38.544 INFO:journalctl@ceph.mon.smithi045.smithi045.stdout:Feb 19 22:26:38 smithi045 podman[44003]: ceph-f6020aec-cf73-11ee-95bb-87774f69a715-mon.smithi045 2024-02-19T22:26:38.548 DEBUG:teuthology.orchestra.run.smithi045:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi045.service' 2024-02-19T22:26:38.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:26:38.646 INFO:tasks.cephadm.mon.smithi045:Stopped mon.smithi045 2024-02-19T22:26:38.646 INFO:tasks.cephadm.mon.smithi097:Stopping mon.smithi097... 2024-02-19T22:26:38.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097 2024-02-19T22:26:38.681 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097.service: Unit ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097.service not loaded. 2024-02-19T22:26:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:26:38.683 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi097" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097' 2024-02-19T22:26:38.684 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:26:38.686 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/ubuntu@smithi045.front.sepia.ceph.com/crash 2024-02-19T22:26:38.687 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:26:38.733 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/ubuntu@smithi097.front.sepia.ceph.com/crash 2024-02-19T22:26:38.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:26:38.775 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:26:38.775 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:26:38.776 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:26:38.776 DEBUG:teuthology.orchestra.run.smithi045:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:26:38.816 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:26:38.817 DEBUG:teuthology.orchestra.run.smithi045:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:26:38.859 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:26:38.887 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-19T22:26:38.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:26:38.887 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:26:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-19T22:26:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:26:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi045.log 2024-02-19T22:26:38.888 INFO:teuthology.orchestra.run.smithi045.stderr:gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:26:38.889 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:26:38.889 INFO:teuthology.orchestra.run.smithi097.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi045.log: /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi045.bvlamo.log 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr: 88.3% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi045.bvlamo.log: 90.3% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr: 88.8% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:26:38.891 INFO:teuthology.orchestra.run.smithi045.stderr:/var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:26:38.892 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-02-19T22:26:38.892 INFO:teuthology.orchestra.run.smithi097.stderr:real 0m0.013s 2024-02-19T22:26:38.892 INFO:teuthology.orchestra.run.smithi097.stderr:user 0m0.010s 2024-02-19T22:26:38.892 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m0.013s 2024-02-19T22:26:38.893 INFO:teuthology.orchestra.run.smithi045.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:26:38.994 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi045.bvlamo.log: file size changed while zipping 2024-02-19T22:26:38.994 INFO:teuthology.orchestra.run.smithi045.stderr: 92.7% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi045.bvlamo.log.gz 2024-02-19T22:26:39.074 INFO:teuthology.orchestra.run.smithi045.stderr: 90.6% -- replaced with /var/log/ceph/f6020aec-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi045.log.gz 2024-02-19T22:26:39.076 INFO:teuthology.orchestra.run.smithi045.stderr: 2024-02-19T22:26:39.077 INFO:teuthology.orchestra.run.smithi045.stderr:real 0m0.200s 2024-02-19T22:26:39.077 INFO:teuthology.orchestra.run.smithi045.stderr:user 0m0.305s 2024-02-19T22:26:39.077 INFO:teuthology.orchestra.run.smithi045.stderr:sys 0m0.021s 2024-02-19T22:26:39.077 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:26:39.077 DEBUG:teuthology.misc:Transferring archived files from smithi045:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/ubuntu@smithi045.front.sepia.ceph.com/log 2024-02-19T22:26:39.078 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:26:39.208 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/ubuntu@smithi097.front.sepia.ceph.com/log 2024-02-19T22:26:39.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:26:39.241 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:26:39.242 DEBUG:teuthology.orchestra.run.smithi045:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f6020aec-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:26:42.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f6020aec-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:29:07.372 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:29:07.373 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:29:07.397 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:29:07.417 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:29:07.417 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097' 2024-02-19T22:29:07.666 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=1748ffabeb08472f9d97f12c1246808d Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097' 2024-02-19T22:29:07.671 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:29:07.682 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:29:07.690 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:29:07.690 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:29:07.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:29:07.816 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:29:07.825 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:29:07.826 DEBUG:teuthology.orchestra.run.smithi045:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:29:07.828 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:29:07.846 INFO:teuthology.orchestra.run.smithi045.stderr:bash: ntpq: command not found 2024-02-19T22:29:07.846 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2024-02-19T22:29:07.852 INFO:teuthology.orchestra.run.smithi045.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:29:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:=============================================================================== 2024-02-19T22:29:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +1935us[+1940us] +/- 107ms 2024-02-19T22:29:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -1613us[-1608us] +/- 60ms 2024-02-19T22:29:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +998us[+1003us] +/- 92ms 2024-02-19T22:29:07.853 INFO:teuthology.orchestra.run.smithi045.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1913us[+1913us] +/- 107ms 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 64 -1633us[-1641us] +/- 59ms 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +990us[ +990us] +/- 93ms 2024-02-19T22:29:07.854 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:29:07.855 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:29:07.864 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:29:07.865 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:29:07.878 DEBUG:teuthology.orchestra.run.smithi045:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:29:07.962 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:29:08.031 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:29:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:29:08.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:29:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:29:08.096 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:29:08.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:29:08.121 INFO:teuthology.task.internal:Duration was 1598.962253 seconds 2024-02-19T22:29:08.122 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:29:08.132 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:29:08.133 DEBUG:teuthology.orchestra.run.smithi045:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:29:08.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:29:08.182 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:29:08.184 INFO:teuthology.orchestra.run.smithi045.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:29:08.422 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:29:08.422 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi045.front.sepia.ceph.com 2024-02-19T22:29:08.422 DEBUG:teuthology.orchestra.run.smithi045:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:29:08.447 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-02-19T22:29:08.448 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:29:08.470 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:29:08.470 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:29:08.490 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:29:08.559 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:29:08.560 DEBUG:teuthology.orchestra.run.smithi045:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:29:08.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:29:08.653 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:29:08.663 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:29:08.663 DEBUG:teuthology.orchestra.run.smithi045:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:29:08.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:29:08.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:29:08.729 DEBUG:teuthology.orchestra.run.smithi045:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:29:08.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:29:08.758 INFO:teuthology.orchestra.run.smithi045.stdout:kernel.core_pattern = core 2024-02-19T22:29:08.784 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-02-19T22:29:08.800 DEBUG:teuthology.orchestra.run.smithi045:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:29:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:29:08.845 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:29:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:29:08.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:29:08.874 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:29:08.875 DEBUG:teuthology.misc:Transferring archived files from smithi045:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/smithi045 2024-02-19T22:29:08.876 DEBUG:teuthology.orchestra.run.smithi045:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:29:08.926 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809/remote/smithi097 2024-02-19T22:29:08.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:29:08.967 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:29:08.967 DEBUG:teuthology.orchestra.run.smithi045:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:29:08.969 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:29:09.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:29:09.030 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:29:09.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:29:09.042 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:29:09.042 DEBUG:teuthology.orchestra.run.smithi045:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:29:09.045 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:29:09.061 INFO:teuthology.orchestra.run.smithi045.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:29 /home/ubuntu/cephtest 2024-02-19T22:29:09.076 INFO:teuthology.orchestra.run.smithi097.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:29 /home/ubuntu/cephtest 2024-02-19T22:29:09.077 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:29:09.088 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:29:09.112 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:29:09.146 DEBUG:teuthology.nuke:shortname: smithi045 2024-02-19T22:29:09.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:29:09.190 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi045.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:53:48.744013', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIp0R+lKPi5Uh/zVH3WN/rPCPMWSYYk6gpVFwEE75Xpm+b2x4NQfSsvXV5SrA97Wg6YP70DCqbujCwosICzA5ME='} 2024-02-19T22:29:09.202 DEBUG:teuthology.nuke:shortname: smithi097 2024-02-19T22:29:09.203 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:29:09.235 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:27:49-fs-pacific-release-distro-default-smithi/7566809', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:53:48.745429', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOmcWGiaGEhl8s/CNIY8wfTWhO4vF2x9vwAaa5jojaK/jtOXiUwr1XR+2PDzNafYJgYVgu1nKXq2ouSD3RY3SBI='} 2024-02-19T22:29:09.284 INFO:teuthology.orchestra.console.smithi045:Power off 2024-02-19T22:29:09.285 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:29:09.340 INFO:teuthology.orchestra.console.smithi097:Power off 2024-02-19T22:29:09.340 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:29:09.350 DEBUG:teuthology.orchestra.console.smithi045:power off output: Chassis Power Control: Down/Off 2024-02-19T22:29:09.350 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:09.389 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-02-19T22:29:09.390 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:09.401 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-02-19T22:29:09.425 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-02-19T22:29:13.403 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:13.426 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:13.515 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is on 2024-02-19T22:29:13.537 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-02-19T22:29:17.520 DEBUG:teuthology.orchestra.console.smithi045:pexpect command: ipmitool -H smithi045.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:17.538 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:17.630 DEBUG:teuthology.orchestra.console.smithi045:check power output: Chassis Power is off 2024-02-19T22:29:17.649 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-02-19T22:29:17.732 INFO:teuthology.orchestra.console.smithi045:Power off completed 2024-02-19T22:29:21.652 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:29:21.762 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-02-19T22:29:21.863 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-02-19T22:29:21.969 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1598.962253332138 failure_reason: 'Command failed on smithi097 with status 5: ''sudo systemctl stop ceph-f6020aec-cf73-11ee-95bb-87774f69a715@mon.smithi097''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=1748ffabeb08472f9d97f12c1246808d status: fail success: false 2024-02-19T22:29:21.969 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:29:22.067 INFO:teuthology.run:FAIL