2024-02-19T21:51:52.402 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T21:51:52.402 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796 --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/orig.config.yaml 2024-02-19T21:51:52.451 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T21:51:52.541 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796 branch: pacific-release description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7566796' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(PG_DEGRADED\) - Degraded data redundancy - \(PG_ - MDS_INSUFFICIENT_STANDBY - deprecated feature inline_data - compat changed unexpectedly - overall HEALTH_ - OSD_DOWN - OSD_ - but it is still running - is not responding sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 264 sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJXcUc/9/OJrMEBUbnTBFhGVF+al70FV2WM6tLlZX8j5d9oYSE+bcqUD9msMfgrZwmtQddO0HbJsYCjMn3tO1Q8= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrkOGisSEJpUGGy6TtGBDenHntsMT3eWmS78WfWD0/XFkXwbIom7RWNLVikoSzy9juQ+rAqsmg2gFTfInoqAXs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_19:25:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T21:51:52.541 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa; will attempt to use it 2024-02-19T21:51:52.542 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks 2024-02-19T21:51:52.542 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T21:51:52.543 INFO:teuthology.task.internal:Checking packages... 2024-02-19T21:51:52.567 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '81bd20d634209c7cb82c18be12b4b5a05643ebf1' 2024-02-19T21:51:52.567 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T21:51:52.567 INFO:teuthology.packaging:ref: None 2024-02-19T21:51:52.567 INFO:teuthology.packaging:tag: None 2024-02-19T21:51:52.567 INFO:teuthology.packaging:branch: pacific-release 2024-02-19T21:51:52.567 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T21:51:52.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-02-19T21:51:52.710 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-548.g81bd20d6 2024-02-19T21:51:52.712 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T21:51:52.719 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T21:51:52.719 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T21:51:52.778 INFO:teuthology.task.internal:Saving configuration 2024-02-19T21:51:52.797 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T21:51:52.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T21:51:52.850 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:02.177237', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJXcUc/9/OJrMEBUbnTBFhGVF+al70FV2WM6tLlZX8j5d9oYSE+bcqUD9msMfgrZwmtQddO0HbJsYCjMn3tO1Q8='} 2024-02-19T21:51:52.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:02.179344', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrkOGisSEJpUGGy6TtGBDenHntsMT3eWmS78WfWD0/XFkXwbIom7RWNLVikoSzy9juQ+rAqsmg2gFTfInoqAXs='} 2024-02-19T21:51:52.873 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T21:51:52.879 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T21:51:52.879 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T21:51:52.880 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T21:51:52.950 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa4134a90d0>, signals=[15]) 2024-02-19T21:51:52.951 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T21:51:52.958 INFO:teuthology.task.internal:Opening connections... 2024-02-19T21:51:52.958 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-02-19T21:51:52.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:53.032 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-02-19T21:51:53.033 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:51:53.103 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T21:51:53.111 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-02-19T21:51:53.133 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-02-19T21:51:53.133 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:51:53.194 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:51:53.195 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-02-19T21:51:53.225 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-02-19T21:51:53.247 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-02-19T21:51:53.248 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 8" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-02-19T21:51:53.309 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-02-19T21:51:53.310 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T21:51:53.310 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-02-19T21:51:53.310 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T21:51:53.310 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-02-19T21:51:53.333 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T21:51:53.346 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T21:51:53.353 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T21:51:53.354 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:51:53.356 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-02-19T21:51:53.377 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T21:51:53.383 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T21:51:53.383 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:51:53.417 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-02-19T21:51:53.444 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T21:51:53.501 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T21:51:53.517 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T21:51:53.517 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T21:51:53.517 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:51:53.518 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T21:51:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:51:53.538 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-02-19T21:51:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:51:53.539 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-02-19T21:51:53.599 INFO:teuthology.orchestra.run.smithi203.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:51:53.599 INFO:teuthology.task.kernel:Running kernel on smithi203: 4.18.0-489.el8.x86_64 2024-02-19T21:51:53.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-02-19T21:51:53.600 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-489.el8.x86_64 2024-02-19T21:51:53.600 INFO:teuthology.task.kernel:Running kernel on smithi111: 4.18.0-489.el8.x86_64 2024-02-19T21:51:53.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-02-19T21:52:05.512 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 2.9 MB/s | 27 MB 00:09 2024-02-19T21:52:06.568 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - AppStream 2.6 MB/s | 27 MB 00:10 2024-02-19T21:52:11.054 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 10 MB 00:00 2024-02-19T21:52:11.985 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - BaseOS 18 MB/s | 10 MB 00:00 2024-02-19T21:52:12.970 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 55 kB/s | 18 kB 00:00 2024-02-19T21:52:13.710 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - Extras 152 kB/s | 18 kB 00:00 2024-02-19T21:52:13.961 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - Extras common packages 49 kB/s | 6.9 kB 00:00 2024-02-19T21:52:14.238 INFO:teuthology.orchestra.run.smithi203.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2024-02-19T21:52:14.555 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 4.7 kB/s | 6.9 kB 00:01 2024-02-19T21:52:14.883 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2024-02-19T21:52:16.297 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2024-02-19T21:52:16.848 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 8.6 MB/s | 16 MB 00:01 2024-02-19T21:52:20.113 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 387 kB/s | 24 kB 00:00 2024-02-19T21:52:20.640 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 442 kB/s | 24 kB 00:00 2024-02-19T21:52:23.110 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:52:23.110 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:52:23.238 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T21:52:23.239 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T21:52:23.239 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:23.239 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T21:52:23.239 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout:Install 3 Packages 2024-02-19T21:52:23.240 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:52:23.241 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 90 M 2024-02-19T21:52:23.241 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 97 M 2024-02-19T21:52:23.241 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T21:52:23.537 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-02-19T21:52:23.537 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-02-19T21:52:23.645 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T21:52:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T21:52:23.646 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T21:52:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T21:52:23.646 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 4.18.0-539.el8 baseos 10 M 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 4.18.0-539.el8 baseos 43 M 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 4.18.0-539.el8 baseos 36 M 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout:Install 3 Packages 2024-02-19T21:52:23.647 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:52:23.648 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 90 M 2024-02-19T21:52:23.648 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 97 M 2024-02-19T21:52:23.648 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T21:52:23.973 INFO:teuthology.orchestra.run.smithi203.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2024-02-19T21:52:24.650 INFO:teuthology.orchestra.run.smithi111.stdout:(1/3): kernel-4.18.0-539.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2024-02-19T21:52:25.715 INFO:teuthology.orchestra.run.smithi203.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-19T21:52:26.217 INFO:teuthology.orchestra.run.smithi111.stdout:(2/3): kernel-modules-4.18.0-539.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2024-02-19T21:52:26.248 INFO:teuthology.orchestra.run.smithi203.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2024-02-19T21:52:26.248 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:26.248 INFO:teuthology.orchestra.run.smithi203.stdout:Total 30 MB/s | 90 MB 00:03 2024-02-19T21:52:26.675 INFO:teuthology.orchestra.run.smithi111.stdout:(3/3): kernel-core-4.18.0-539.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2024-02-19T21:52:26.676 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T21:52:26.676 INFO:teuthology.orchestra.run.smithi111.stdout:Total 30 MB/s | 90 MB 00:03 2024-02-19T21:52:26.934 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T21:52:27.352 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T21:52:27.355 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T21:52:27.355 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T21:52:27.767 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T21:52:27.767 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T21:52:28.607 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T21:52:28.627 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T21:52:28.995 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T21:52:29.023 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T21:52:31.391 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T21:52:31.797 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T21:52:31.869 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:32.290 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:35.485 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:35.891 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:52:35.894 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:36.298 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:39.337 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:39.668 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:52:39.735 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:52:40.132 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:01.992 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.014 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.286 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.309 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.607 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.608 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:53:02.608 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:53:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-4.18.0-539.el8.x86_64 1/3 2024-02-19T21:53:02.683 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-4.18.0-539.el8.x86_64 2/3 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:53:03.102 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-4.18.0-539.el8.x86_64 3/3 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-4.18.0-539.el8.x86_64 kernel-core-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:53:03.183 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T21:53:03.245 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:03.320 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:03.464 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.465 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.465 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.465 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:53:03.465 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:53:03.465 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T21:53:03.465 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:53:03.465 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:03.539 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.540 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.540 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.540 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T21:53:03.540 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T21:53:03.540 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T21:53:03.540 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T21:53:03.540 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T21:53:03.676 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.676 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.676 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.02 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 129.el8 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Group : System Environment/Base 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 9265586 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-02-19T21:53:03.710 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Relocations : (not relocatable) 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : CentOS BuildSystem 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:53:03.711 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:53:03.712 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:53:03.712 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-02-19T21:53:03.712 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T21:53:03.712 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:53:03.713 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:53:03.713 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.713 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:53:03.751 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-4.18.0-539.el8.x86_64 2024-02-19T21:53:03.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.751 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.02 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 129.el8 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Group : System Environment/Base 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 9265586 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : kbuilder.bsys.centos.org 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Relocations : (not relocatable) 2024-02-19T21:53:03.784 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : CentOS BuildSystem 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T21:53:03.785 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T21:53:03.786 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T21:53:03.786 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 4.18.0-539.el8.x86_64 2024-02-19T21:53:03.786 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T21:53:05.030 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-02-19T21:53:05.453 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-02-19T21:53:07.427 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-02-19T21:53:07.436 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-02-19T21:53:07.470 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.Ot8BgexBeW 2024-02-19T21:53:07.470 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ot8BgexBeW 2024-02-19T21:53:07.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.Ot8BgexBeW 2024-02-19T21:53:07.686 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.Ot8BgexBeW is 6KB 2024-02-19T21:53:07.734 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.Ot8BgexBeW 2024-02-19T21:53:07.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:53:07.823 INFO:teuthology.orchestra.run.smithi203.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:53:07.823 INFO:teuthology.orchestra.run.smithi203.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:53:07.823 INFO:teuthology.orchestra.run.smithi203.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:53:07.823 INFO:teuthology.orchestra.run.smithi203.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:53:07.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:53:07.999 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-02-19T21:53:08.198 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-02-19T21:53:08.200 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-02-19T21:53:08.230 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.fRQ0CRWpdX 2024-02-19T21:53:08.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fRQ0CRWpdX 2024-02-19T21:53:08.304 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.fRQ0CRWpdX 2024-02-19T21:53:08.445 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.fRQ0CRWpdX is 6KB 2024-02-19T21:53:08.466 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.fRQ0CRWpdX 2024-02-19T21:53:08.483 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T21:53:08.557 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-02-19T21:53:08.558 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-02-19T21:53:08.558 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-02-19T21:53:08.558 INFO:teuthology.orchestra.run.smithi111.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64.conf 2024-02-19T21:53:08.559 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-539.el8.x86_64 2024-02-19T21:53:08.688 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-02-19T21:53:38.028 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T21:53:38.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T21:53:38.029 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:53:38.691 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T21:53:38.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:53:38.692 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:53:56.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-02-19T21:53:57.052 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-19T21:54:05.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T21:54:05.549 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:06.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:54:06.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:08.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-02-19T21:54:09.116 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-19T21:54:20.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T21:54:20.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:21.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:54:21.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:24.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-19T21:54:39.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-02-19T21:54:39.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:54:39.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:54.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T21:54:54.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:54:54.418 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-02-19T21:54:54.738 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T21:54:54.738 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:54:54.738 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-02-19T21:54:54.795 INFO:teuthology.orchestra.run.smithi203.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:54:54.795 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:54:54.795 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:54:54.795 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:54:55.796 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T21:54:55.796 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:54:55.884 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-02-19T21:54:55.896 DEBUG:teuthology.parallel:result is None 2024-02-19T21:54:57.727 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-02-19T21:55:15.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:55:15.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T21:55:16.063 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-02-19T21:55:16.362 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T21:55:16.362 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-539.el8.x86_64"... 2024-02-19T21:55:16.362 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-02-19T21:55:16.383 INFO:teuthology.orchestra.run.smithi111.stdout:4.18.0-539.el8.x86_64 2024-02-19T21:55:16.383 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-539.el8.x86_64 vs 4.18.0-539.el8.x86_64 2024-02-19T21:55:16.383 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T21:55:16.383 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T21:55:17.384 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T21:55:17.384 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T21:55:17.462 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-02-19T21:55:17.478 DEBUG:teuthology.parallel:result is None 2024-02-19T21:55:17.479 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T21:55:17.486 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T21:55:17.486 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:55:17.489 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T21:55:17.513 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T21:55:17.519 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T21:55:17.528 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T21:55:17.529 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:55:17.550 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T21:55:17.602 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T21:55:17.609 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T21:55:17.610 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:55:17.628 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T21:55:17.663 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:17.671 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:17.674 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:17.682 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T21:55:17.684 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T21:55:17.691 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T21:55:17.692 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:55:17.718 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T21:55:17.754 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T21:55:17.767 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T21:55:17.767 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:55:17.788 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T21:55:17.813 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:17.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:17.944 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:18.029 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:18.073 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T21:55:18.073 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:55:18.140 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:18.189 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T21:55:18.245 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:18.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T21:55:18.375 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T21:55:18.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T21:55:18.443 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-02-19T21:55:18.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-02-19T21:55:18.491 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:55:18.557 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T21:55:18.969 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T21:55:18.978 INFO:teuthology.task.internal:Starting timer... 2024-02-19T21:55:18.978 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T21:55:18.987 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T21:55:18.996 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T21:55:18.996 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-02-19T21:55:19.136 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-02-19T21:55:19.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-02-19T21:55:19.198 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-02-19T21:55:19.199 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T21:55:19.199 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-02-19T21:55:19.236 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-02-19T21:55:19.237 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-02-19T21:55:19.298 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-02-19T21:55:19.298 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-02-19T21:55:19.299 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:55:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:55:19.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T21:55:19.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T21:55:19.354 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T21:55:19.354 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:55:19.395 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-02-19T21:55:19.442 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T21:55:19.453 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-19T21:55:19.459 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-19T21:55:19.590 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T21:55:19.607 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T21:55:19.609 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T22:01:18.351 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-02-19T22:01:18.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T22:01:18.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:18.427 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-02-19T22:01:18.496 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-02-19T22:01:18.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T22:01:18.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T22:01:18.567 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-02-19T22:01:18.634 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-02-19T22:01:18.634 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T22:01:18.646 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T22:01:18.646 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:01:18.647 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:01:18.649 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T22:01:18.649 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:01:18.680 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:01:18.695 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:01:18.705 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T22:01:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T22:01:18.766 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-02-19T22:01:18.776 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-02-19T22:01:18.786 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-02-19T22:01:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:01:18.798 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-02-19T22:01:18.803 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:01:18.811 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T22:01:18.824 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T22:01:18.875 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.878 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.896 INFO:teuthology.orchestra.run.smithi203.stderr:bash: ntpq: command not found 2024-02-19T22:01:18.899 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:01:18.899 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-02-19T22:01:18.900 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.900 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.900 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.900 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:01:18.901 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:01:18.911 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:01:18.911 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2024-02-19T22:01:18.911 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:18.911 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi203.front.sepia.ceph.com 2024-02-19T22:01:18.911 DEBUG:teuthology.orchestra.run.smithi203:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:19.703 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 09:59:44 PM UTC. 2024-02-19T22:01:19.728 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 09:59:44 PM UTC. 2024-02-19T22:01:21.705 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:01:21.706 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:21.706 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:01:21.706 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:21.706 INFO:teuthology.orchestra.run.smithi111.stdout:Resetting modules: 2024-02-19T22:01:21.706 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools 2024-02-19T22:01:21.707 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:21.707 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:01:21.707 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:21.707 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:21.735 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:Resetting modules: 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout: container-tools 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:21.736 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:21.959 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:01:21.988 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:01:22.806 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 09:59:44 PM UTC. 2024-02-19T22:01:22.851 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 09:59:44 PM UTC. 2024-02-19T22:01:24.860 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:01:24.887 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout:Installing group/module packages: 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:01:24.895 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout:Installing weak dependencies: 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout:Installing module profiles: 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools/common 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout:Enabling module streams: 2024-02-19T22:01:24.896 INFO:teuthology.orchestra.run.smithi111.stdout: container-tools rhel8 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 28 M 2024-02-19T22:01:24.897 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 92 M 2024-02-19T22:01:24.898 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout:Installing group/module packages: 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: aardvark-dns x86_64 2:1.10.0-1.module_el8+871+552be958 appstream 1.0 M 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: buildah x86_64 2:1.33.5-1.module_el8+885+7da147f3 appstream 9.7 M 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: cockpit-podman noarch 83-1.module_el8+847+7863d4e6 appstream 688 k 2024-02-19T22:01:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: crun x86_64 1.14-1.module_el8+871+552be958 appstream 257 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: netavark x86_64 2:1.10.2-1.module_el8+885+7da147f3 appstream 4.3 M 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: python3-podman noarch 4.9.0-1.module_el8+847+7863d4e6 appstream 180 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: skopeo x86_64 2:1.14.3-0.1.module_el8+885+7da147f3 appstream 8.8 M 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: toolbox x86_64 0.0.99.5-1.module_el8+847+7863d4e6 appstream 2.5 M 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: udica noarch 0.2.6-20.module_el8+804+f131391c appstream 53 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout:Installing module profiles: 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: container-tools/common 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:24.924 INFO:teuthology.orchestra.run.smithi203.stdout:Enabling module streams: 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout: container-tools rhel8 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:24.925 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 28 M 2024-02-19T22:01:24.926 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 92 M 2024-02-19T22:01:24.926 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:01:25.486 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.9 MB/s | 688 kB 00:00 2024-02-19T22:01:25.561 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): aardvark-dns-1.10.0-1.module_el8+871+55 2.3 MB/s | 1.0 MB 00:00 2024-02-19T22:01:25.586 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 378 kB/s | 38 kB 00:00 2024-02-19T22:01:25.651 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): cockpit-podman-83-1.module_el8+847+7863 1.4 MB/s | 688 kB 00:00 2024-02-19T22:01:25.661 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): crun-1.14-1.module_el8+871+552be958.x86 2.5 MB/s | 257 kB 00:00 2024-02-19T22:01:25.726 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 506 kB/s | 38 kB 00:00 2024-02-19T22:01:25.787 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 1.4 MB/s | 180 kB 00:00 2024-02-19T22:01:25.845 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 426 kB/s | 25 kB 00:00 2024-02-19T22:01:25.901 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): crun-1.14-1.module_el8+871+552be958.x86 1.4 MB/s | 257 kB 00:00 2024-02-19T22:01:25.912 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2024-02-19T22:01:26.285 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): aardvark-dns-1.10.0-1.module_el8+871+55 914 kB/s | 1.0 MB 00:01 2024-02-19T22:01:26.477 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-podman-4.9.0-1.module_el8+847+7 941 kB/s | 180 kB 00:00 2024-02-19T22:01:26.544 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 372 kB/s | 25 kB 00:00 2024-02-19T22:01:26.694 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 626 kB/s | 94 kB 00:00 2024-02-19T22:01:28.688 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.4 MB/s | 4.3 MB 00:03 2024-02-19T22:01:30.003 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): netavark-1.10.2-1.module_el8+885+7da147 1.1 MB/s | 4.3 MB 00:04 2024-02-19T22:01:30.538 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): buildah-1.33.5-1.module_el8+885+7da147f 1.8 MB/s | 9.7 MB 00:05 2024-02-19T22:01:30.596 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 908 kB/s | 53 kB 00:00 2024-02-19T22:01:30.663 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): toolbox-0.0.99.5-1.module_el8+847+7863 1.3 MB/s | 2.5 MB 00:01 2024-02-19T22:01:30.697 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 409 kB/s | 41 kB 00:00 2024-02-19T22:01:31.953 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): toolbox-0.0.99.5-1.module_el8+847+7863d 1.3 MB/s | 2.5 MB 00:01 2024-02-19T22:01:32.053 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): udica-0.2.6-20.module_el8+804+f131391c 527 kB/s | 53 kB 00:00 2024-02-19T22:01:32.112 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 705 kB/s | 41 kB 00:00 2024-02-19T22:01:34.972 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 995 kB/s | 8.8 MB 00:09 2024-02-19T22:01:34.972 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:01:34.972 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.7 MB/s | 28 MB 00:10 2024-02-19T22:01:35.037 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): buildah-1.33.5-1.module_el8+885+7da147 1.0 MB/s | 9.7 MB 00:09 2024-02-19T22:01:35.245 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:01:35.266 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:01:35.266 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:01:35.550 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:01:35.553 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:01:36.238 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:01:36.487 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): skopeo-1.14.3-0.1.module_el8+885+7da14 921 kB/s | 8.8 MB 00:09 2024-02-19T22:01:36.488 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:01:36.488 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.4 MB/s | 28 MB 00:11 2024-02-19T22:01:36.761 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:01:36.782 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:01:36.782 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:01:37.049 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:01:37.052 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:01:37.388 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:01:37.606 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:01:37.785 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:01:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:01:37.984 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:01:38.228 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:01:38.789 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-02-19T22:01:38.807 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:01:39.005 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:01:39.167 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/13 2024-02-19T22:01:39.418 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 4/13 2024-02-19T22:01:39.652 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 5/13 2024-02-19T22:01:39.771 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:01:40.012 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:01:40.240 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:01:40.256 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 6/13 2024-02-19T22:01:40.706 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:01:40.970 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:01:41.180 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:01:41.204 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 7/13 2024-02-19T22:01:41.384 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:01:41.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 8/13 2024-02-19T22:01:41.623 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : crun-1.14-1.module_el8+871+552be958.x86_64 9/13 2024-02-19T22:01:41.800 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:01:41.800 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:01:41.800 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:01:41.800 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:01:41.800 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:01:41.801 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:01:41.802 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:01:41.802 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:01:42.120 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 10/13 2024-02-19T22:01:42.351 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:01:42.463 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:01:42.464 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:01:42.465 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:01:42.555 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:01:42.758 DEBUG:teuthology.parallel:result is None 2024-02-19T22:01:42.799 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 13/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x8 1/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 3/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : crun-1.14-1.module_el8+871+552be958.x86_64 5/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 6/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-podman-4.9.0-1.module_el8+847+7863d4e6.noa 7/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 10/13 2024-02-19T22:01:43.223 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 11/13 2024-02-19T22:01:43.224 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : udica-0.2.6-20.module_el8+804+f131391c.noarch 12/13 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: aardvark-dns-2:1.10.0-1.module_el8+871+552be958.x86_64 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: buildah-2:1.33.5-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: cockpit-podman-83-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:01:43.807 INFO:teuthology.orchestra.run.smithi203.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: crun-1.14-1.module_el8+871+552be958.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: netavark-2:1.10.2-1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: python3-podman-4.9.0-1.module_el8+847+7863d4e6.noarch 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: skopeo-2:1.14.3-0.1.module_el8+885+7da147f3.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: toolbox-0.0.99.5-1.module_el8+847+7863d4e6.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: udica-0.2.6-20.module_el8+804+f131391c.noarch 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: yajl-2.1.0-12.el8.x86_64 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:01:43.808 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:01:44.087 DEBUG:teuthology.parallel:result is None 2024-02-19T22:01:44.088 INFO:teuthology.run_tasks:Running task pexec... 2024-02-19T22:01:44.096 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-19T22:01:44.096 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi111.front.sepia.ceph.com 2024-02-19T22:01:44.097 DEBUG:teuthology.orchestra.run.smithi111:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:44.097 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi203.front.sepia.ceph.com 2024-02-19T22:01:44.097 DEBUG:teuthology.orchestra.run.smithi203:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-19T22:01:44.258 DEBUG:teuthology.parallel:result is None 2024-02-19T22:01:44.276 DEBUG:teuthology.parallel:result is None 2024-02-19T22:01:44.277 INFO:teuthology.run_tasks:Running task install... 2024-02-19T22:01:44.285 DEBUG:teuthology.task.install:project ceph 2024-02-19T22:01:44.285 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'}} 2024-02-19T22:01:44.285 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:01:44.285 INFO:teuthology.task.install:Using flavor: default 2024-02-19T22:01:44.293 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T22:01:44.293 INFO:teuthology.task.install:extra packages: [] 2024-02-19T22:01:44.293 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:01:44.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:01:44.294 INFO:teuthology.packaging:ref: None 2024-02-19T22:01:44.294 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:01:44.294 INFO:teuthology.packaging:branch: None 2024-02-19T22:01:44.294 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:01:44.483 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:01:44.484 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:01:44.487 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-02-19T22:01:44.487 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:01:44.487 INFO:teuthology.packaging:ref: None 2024-02-19T22:01:44.487 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:01:44.487 INFO:teuthology.packaging:branch: None 2024-02-19T22:01:44.487 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:01:44.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:01:44.711 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:01:44.711 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:01:44.714 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-02-19T22:01:44.714 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-02-19T22:01:44.751 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:01:44.752 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:01:44.752 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:01:44.761 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T22:01:44.761 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:01:44.762 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T22:01:44.784 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:01:44.784 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:01:44.784 INFO:teuthology.packaging:ref: None 2024-02-19T22:01:44.784 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:01:44.785 INFO:teuthology.packaging:branch: None 2024-02-19T22:01:44.785 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:01:44.785 DEBUG:teuthology.orchestra.run.smithi111:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:01:44.789 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T22:01:44.789 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:01:44.789 INFO:teuthology.packaging:ref: None 2024-02-19T22:01:44.789 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:01:44.790 INFO:teuthology.packaging:branch: None 2024-02-19T22:01:44.790 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:01:44.790 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T22:01:44.856 DEBUG:teuthology.orchestra.run.smithi111:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:01:44.857 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T22:01:44.933 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:01:44.935 DEBUG:teuthology.orchestra.run.smithi111:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T22:01:45.000 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-02-19T22:01:45.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-02-19T22:01:45.003 INFO:teuthology.orchestra.run.smithi111.stdout:check_obsoletes = 1 2024-02-19T22:01:45.005 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum clean all 2024-02-19T22:01:45.268 INFO:teuthology.orchestra.run.smithi111.stdout:76 files removed 2024-02-19T22:01:45.274 INFO:teuthology.orchestra.run.smithi203.stdout:76 files removed 2024-02-19T22:01:45.296 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-radosgw 2024-02-19T22:01:45.305 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-02-19T22:01:45.781 INFO:teuthology.orchestra.run.smithi111.stdout:ceph packages for x86_64 922 kB/s | 74 kB 00:00 2024-02-19T22:01:45.782 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 1.0 MB/s | 74 kB 00:00 2024-02-19T22:01:45.883 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 242 kB/s | 15 kB 00:00 2024-02-19T22:01:45.890 INFO:teuthology.orchestra.run.smithi111.stdout:ceph noarch packages 213 kB/s | 15 kB 00:00 2024-02-19T22:01:45.984 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2024-02-19T22:01:46.015 INFO:teuthology.orchestra.run.smithi111.stdout:ceph source packages 18 kB/s | 1.7 kB 00:00 2024-02-19T22:02:04.889 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - AppStream 1.4 MB/s | 27 MB 00:18 2024-02-19T22:02:04.982 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - AppStream 1.4 MB/s | 27 MB 00:18 2024-02-19T22:02:10.792 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - BaseOS 11 MB/s | 10 MB 00:00 2024-02-19T22:02:11.282 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - BaseOS 6.7 MB/s | 10 MB 00:01 2024-02-19T22:02:12.667 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - Extras 62 kB/s | 18 kB 00:00 2024-02-19T22:02:13.084 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2024-02-19T22:02:13.216 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2024-02-19T22:02:13.491 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - Extras common packages 42 kB/s | 6.9 kB 00:00 2024-02-19T22:02:14.472 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 8 - PowerTools 3.7 MB/s | 4.8 MB 00:01 2024-02-19T22:02:14.910 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 8 - PowerTools 3.7 MB/s | 4.8 MB 00:01 2024-02-19T22:02:15.722 INFO:teuthology.orchestra.run.smithi203.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2024-02-19T22:02:16.161 INFO:teuthology.orchestra.run.smithi111.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2024-02-19T22:02:16.772 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-02-19T22:02:20.556 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 416 kB/s | 24 kB 00:00 2024-02-19T22:02:24.053 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:02:24.056 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:02:24.057 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:02:24.058 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout:Install 25 Packages 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-02-19T22:02:24.059 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:02:24.060 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 59 M 2024-02-19T22:02:24.060 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:02:24.391 INFO:teuthology.orchestra.run.smithi203.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 52 MB/s | 6.0 MB 00:00 2024-02-19T22:02:24.416 INFO:teuthology.orchestra.run.smithi203.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2024-02-19T22:02:24.458 INFO:teuthology.orchestra.run.smithi203.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2024-02-19T22:02:24.492 INFO:teuthology.orchestra.run.smithi203.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-02-19T22:02:24.593 INFO:teuthology.orchestra.run.smithi203.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2024-02-19T22:02:24.609 INFO:teuthology.orchestra.run.smithi203.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 2.6 MB/s | 44 kB 00:00 2024-02-19T22:02:24.634 INFO:teuthology.orchestra.run.smithi203.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T22:02:24.660 INFO:teuthology.orchestra.run.smithi203.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.3 MB/s | 213 kB 00:00 2024-02-19T22:02:24.693 INFO:teuthology.orchestra.run.smithi203.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-19T22:02:24.727 INFO:teuthology.orchestra.run.smithi203.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2024-02-19T22:02:24.752 INFO:teuthology.orchestra.run.smithi203.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-02-19T22:02:24.920 INFO:teuthology.orchestra.run.smithi203.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-02-19T22:02:24.970 INFO:teuthology.orchestra.run.smithi203.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 669 kB/s | 145 kB 00:00 2024-02-19T22:02:25.229 INFO:teuthology.orchestra.run.smithi203.stdout:(14/27): ceph-common-16.2.4-0.el8.x86_64.rpm 25 MB/s | 24 MB 00:00 2024-02-19T22:02:25.254 INFO:teuthology.orchestra.run.smithi203.stdout:(15/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 916 kB/s | 259 kB 00:00 2024-02-19T22:02:25.337 INFO:teuthology.orchestra.run.smithi203.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2024-02-19T22:02:25.454 INFO:teuthology.orchestra.run.smithi203.stdout:(17/27): daxctl-libs-71.1-7.el8.x86_64.rpm 210 kB/s | 42 kB 00:00 2024-02-19T22:02:25.538 INFO:teuthology.orchestra.run.smithi203.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 235 kB/s | 47 kB 00:00 2024-02-19T22:02:25.572 INFO:teuthology.orchestra.run.smithi203.stdout:(19/27): ndctl-libs-71.1-7.el8.x86_64.rpm 683 kB/s | 80 kB 00:00 2024-02-19T22:02:25.648 INFO:teuthology.orchestra.run.smithi203.stdout:(20/27): librdkafka-1.6.1-1.el8.x86_64.rpm 930 kB/s | 677 kB 00:00 2024-02-19T22:02:25.673 INFO:teuthology.orchestra.run.smithi203.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 866 kB/s | 116 kB 00:00 2024-02-19T22:02:25.706 INFO:teuthology.orchestra.run.smithi203.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 2.2 MB/s | 306 kB 00:00 2024-02-19T22:02:25.732 INFO:teuthology.orchestra.run.smithi203.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2024-02-19T22:02:25.757 INFO:teuthology.orchestra.run.smithi203.stdout:(24/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2024-02-19T22:02:25.782 INFO:teuthology.orchestra.run.smithi203.stdout:(25/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-02-19T22:02:25.874 INFO:teuthology.orchestra.run.smithi203.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 26 MB/s | 3.7 MB 00:00 2024-02-19T22:02:25.933 INFO:teuthology.orchestra.run.smithi203.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2024-02-19T22:02:25.935 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:02:25.935 INFO:teuthology.orchestra.run.smithi203.stdout:Total 31 MB/s | 59 MB 00:01 2024-02-19T22:02:26.004 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:02:26.052 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:02:26.052 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:02:26.400 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:02:26.403 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:02:27.554 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:02:27.779 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:02:28.025 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:02:28.192 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:02:28.605 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:02:28.792 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:02:28.923 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:02:29.180 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:02:29.392 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:02:29.455 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:02:29.702 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:02:30.270 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:02:30.450 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:02:30.528 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:02:30.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:02:30.917 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:02:30.985 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:02:31.139 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:02:31.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:02:31.509 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:02:31.578 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:02:31.776 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:02:32.187 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:02:32.400 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:02:32.501 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:02:32.687 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:02:32.935 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:02:33.092 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:02:33.165 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:02:33.392 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:02:33.570 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:02:33.791 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:02:34.000 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:02:36.559 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:36.776 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:37.485 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:02:37.781 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:02:37.887 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:02:38.041 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /dev 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /run 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-02-19T22:02:57.420 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-02-19T22:02:57.421 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-02-19T22:02:57.421 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-02-19T22:02:57.421 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /dev 2024-02-19T22:02:57.421 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:02:58.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:02:58.850 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:02:58.850 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:02:59.040 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:02:59.040 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:02:59.744 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:02:59.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:02:59.746 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.721 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:03:00.722 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:00.723 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:01.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-02-19T22:03:01.794 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:03.339 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:03.340 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:03:03.340 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:03.340 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:03.340 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-02-19T22:03:03.341 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:03.342 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 53 M 2024-02-19T22:03:03.342 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 229 M 2024-02-19T22:03:03.342 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:03.476 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 169 kB 00:00 2024-02-19T22:03:03.526 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.6 MB/s | 203 kB 00:00 2024-02-19T22:03:03.643 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.1 MB/s | 188 kB 00:00 2024-02-19T22:03:03.668 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.2 MB/s | 323 kB 00:00 2024-02-19T22:03:03.744 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 943 kB/s | 71 kB 00:00 2024-02-19T22:03:04.252 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 61 MB/s | 52 MB 00:00 2024-02-19T22:03:04.254 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:04.255 INFO:teuthology.orchestra.run.smithi203.stdout:Total 58 MB/s | 53 MB 00:00 2024-02-19T22:03:04.297 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:04.318 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:04.318 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:04.630 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:04.633 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:03:05.157 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:03:05.423 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:03:05.629 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:03:05.820 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:03:05.882 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:03:06.090 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:03:06.270 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:03:11.123 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:03:11.335 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:03:11.632 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:03:11.632 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:03:11.632 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:03:11.633 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:03:11.633 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:03:11.633 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:12.144 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:12.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-02-19T22:03:13.128 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:53 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:14.673 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:14.676 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:14.676 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:03:14.677 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:03:14.678 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:14.679 INFO:teuthology.orchestra.run.smithi203.stdout:Install 32 Packages 2024-02-19T22:03:14.680 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:14.680 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 M 2024-02-19T22:03:14.680 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 111 M 2024-02-19T22:03:14.680 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:14.917 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 72 kB/s | 6.3 kB 00:00 2024-02-19T22:03:14.974 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-02-19T22:03:15.033 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2024-02-19T22:03:15.058 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-19T22:03:15.150 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-02-19T22:03:15.175 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.3 MB/s | 157 kB 00:00 2024-02-19T22:03:15.217 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.4 MB/s | 103 kB 00:00 2024-02-19T22:03:15.234 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 462 kB/s | 39 kB 00:00 2024-02-19T22:03:15.251 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2024-02-19T22:03:15.493 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 35 MB/s | 18 MB 00:00 2024-02-19T22:03:15.552 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.4 MB/s | 457 kB 00:00 2024-02-19T22:03:15.769 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 480 kB/s | 249 kB 00:00 2024-02-19T22:03:15.803 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 435 kB/s | 109 kB 00:00 2024-02-19T22:03:15.886 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 374 kB/s | 44 kB 00:00 2024-02-19T22:03:15.920 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 1.5 MB/s | 641 kB 00:00 2024-02-19T22:03:15.987 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2024-02-19T22:03:16.012 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2024-02-19T22:03:16.037 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2024-02-19T22:03:16.071 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 299 kB/s | 18 kB 00:00 2024-02-19T22:03:16.097 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 830 kB/s | 49 kB 00:00 2024-02-19T22:03:16.130 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.0 MB/s | 59 kB 00:00 2024-02-19T22:03:16.156 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.6 MB/s | 384 kB 00:00 2024-02-19T22:03:16.181 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2024-02-19T22:03:16.206 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 206 kB/s | 16 kB 00:00 2024-02-19T22:03:16.231 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-19T22:03:16.257 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-19T22:03:16.282 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-02-19T22:03:16.316 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 471 kB/s | 27 kB 00:00 2024-02-19T22:03:16.341 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-02-19T22:03:16.366 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2024-02-19T22:03:16.392 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-02-19T22:03:16.437 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 326 kB/s | 23 kB 00:00 2024-02-19T22:03:16.438 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:16.439 INFO:teuthology.orchestra.run.smithi203.stdout:Total 18 MB/s | 31 MB 00:01 2024-02-19T22:03:16.520 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:16.559 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:16.559 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:16.869 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:16.872 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:03:18.006 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:03:18.250 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:03:18.467 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:03:18.738 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:03:18.984 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:03:19.181 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:03:19.424 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:03:19.604 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:03:19.791 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:03:20.028 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:03:20.240 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:03:20.503 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:03:20.744 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:03:21.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:03:21.481 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:03:21.668 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:03:21.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:03:22.161 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:03:22.514 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:03:22.742 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:03:22.955 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:03:23.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:03:23.465 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:03:23.698 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:03:23.966 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:03:24.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:03:24.649 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:03:25.078 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:03:25.323 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:03:26.862 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:03:27.056 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:03:27.490 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:03:27.722 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:03:27.945 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:03:28.133 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:03:28.151 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:03:28.316 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:03:28.733 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:03:28.734 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:03:28.735 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:03:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:03:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:29.536 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:29.536 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:03:29.537 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:03:29.538 INFO:teuthology.orchestra.run.smithi203.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:29.539 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:29.848 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-02-19T22:03:30.606 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:10 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:32.098 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:03:32.144 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:32.145 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:03:32.145 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:32.248 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-02-19T22:03:33.003 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:13 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:34.583 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:34.584 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:03:34.585 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:34.585 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 71 k 2024-02-19T22:03:34.585 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 290 k 2024-02-19T22:03:34.585 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:34.625 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 71 kB 00:00 2024-02-19T22:03:34.626 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:34.626 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.7 MB/s | 71 kB 00:00 2024-02-19T22:03:34.626 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:34.633 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:34.633 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:34.634 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:34.637 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:03:34.678 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:03:35.193 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:03:35.375 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:03:35.610 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:03:35.989 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:03:35.989 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:35.989 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:35.990 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:03:35.990 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:35.990 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:36.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:03:36.967 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:38.522 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:38.523 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:03:38.523 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:38.523 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 163 k 2024-02-19T22:03:38.523 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 587 k 2024-02-19T22:03:38.524 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:38.948 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 384 kB/s | 163 kB 00:00 2024-02-19T22:03:38.949 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:38.949 INFO:teuthology.orchestra.run.smithi203.stdout:Total 383 kB/s | 163 kB 00:00 2024-02-19T22:03:38.949 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:38.960 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:38.960 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:39.010 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:39.013 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:03:39.126 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:03:39.315 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:03:39.731 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:40.114 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:40.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-02-19T22:03:41.095 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:21 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:42.589 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:03:42.636 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:42.637 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:03:42.637 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:42.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:03:43.497 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:45.022 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:45.023 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:45.023 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:45.024 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:03:45.025 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-02-19T22:03:45.025 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:45.025 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.0 M 2024-02-19T22:03:45.025 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 15 M 2024-02-19T22:03:45.025 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:45.302 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 242 kB/s | 24 kB 00:00 2024-02-19T22:03:45.349 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 61 kB/s | 8.9 kB 00:00 2024-02-19T22:03:45.411 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 13 MB/s | 2.7 MB 00:00 2024-02-19T22:03:45.578 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 154 kB/s | 43 kB 00:00 2024-02-19T22:03:46.362 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 33 kB/s | 34 kB 00:01 2024-02-19T22:03:46.762 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 145 kB/s | 196 kB 00:01 2024-02-19T22:03:46.765 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:46.765 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.7 MB/s | 3.0 MB 00:01 2024-02-19T22:03:46.777 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:46.788 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:46.788 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:46.956 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:46.959 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:03:47.220 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:03:47.503 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:03:47.764 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:03:47.990 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:03:48.119 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:03:49.135 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:03:49.443 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:03:49.721 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:03:49.721 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:03:49.721 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:03:49.721 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:03:49.722 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:03:49.722 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:03:50.112 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:03:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:03:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:03:50.113 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:50.113 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:03:50.336 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:03:51.097 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:31 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:03:52.679 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:03:52.680 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:03:52.681 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout:Install 8 Packages 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-02-19T22:03:52.682 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 210 M 2024-02-19T22:03:52.683 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:03:53.060 INFO:teuthology.orchestra.run.smithi203.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 59 MB/s | 8.6 MB 00:00 2024-02-19T22:03:53.810 INFO:teuthology.orchestra.run.smithi203.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.4 MB/s | 4.9 MB 00:00 2024-02-19T22:03:53.877 INFO:teuthology.orchestra.run.smithi203.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 4.9 MB/s | 4.7 MB 00:00 2024-02-19T22:03:53.919 INFO:teuthology.orchestra.run.smithi203.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-02-19T22:03:54.269 INFO:teuthology.orchestra.run.smithi203.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-02-19T22:03:54.345 INFO:teuthology.orchestra.run.smithi203.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 2.2 MB/s | 172 kB 00:00 2024-02-19T22:03:55.579 INFO:teuthology.orchestra.run.smithi203.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 1.5 MB/s | 3.7 MB 00:02 2024-02-19T22:03:59.305 INFO:teuthology.orchestra.run.smithi203.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.6 MB/s | 14 MB 00:05 2024-02-19T22:03:59.305 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:03:59.306 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.6 MB/s | 37 MB 00:06 2024-02-19T22:03:59.540 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:03:59.558 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:03:59.558 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:03:59.894 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:03:59.897 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:00.660 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:01.228 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:04:01.356 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:04:01.524 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:04:02.105 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:04:02.303 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:02.897 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:03.058 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:04:03.748 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:04:04.139 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:04:04.433 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:04:06.196 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:04:07.526 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:04:07.748 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:04:08.083 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:04:08.083 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:04:08.084 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:04:08.486 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:08.487 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:08.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:04:09.354 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:49 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:10.903 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:04:10.905 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:10.906 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 1.9 M 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 24 M 2024-02-19T22:04:10.907 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:11.239 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 603 kB/s | 27 kB 00:00 2024-02-19T22:04:11.289 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 210 kB/s | 20 kB 00:00 2024-02-19T22:04:11.306 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 238 kB/s | 27 kB 00:00 2024-02-19T22:04:11.340 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2024-02-19T22:04:11.365 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2024-02-19T22:04:11.499 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 247 kB/s | 33 kB 00:00 2024-02-19T22:04:11.541 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 212 kB/s | 43 kB 00:00 2024-02-19T22:04:11.610 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 147 kB/s | 16 kB 00:00 2024-02-19T22:04:11.659 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 439 kB/s | 155 kB 00:00 2024-02-19T22:04:11.750 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 389 kB/s | 81 kB 00:00 2024-02-19T22:04:11.817 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 407 kB/s | 64 kB 00:00 2024-02-19T22:04:11.842 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 665 kB/s | 61 kB 00:00 2024-02-19T22:04:11.876 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.4 MB/s | 1.2 MB 00:00 2024-02-19T22:04:11.876 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:11.876 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-02-19T22:04:11.928 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:11.943 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:11.943 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:12.028 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:12.032 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:12.578 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:12.803 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:04:12.973 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:04:13.228 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:04:13.457 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:04:13.707 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:04:13.949 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:04:14.292 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:04:14.557 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:04:15.592 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:04:15.857 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:04:16.097 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:04:16.307 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:04:16.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:04:16.813 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:04:16.814 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:04:16.814 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:04:16.814 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:04:17.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:04:17.544 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:04:17.545 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:04:17.545 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:17.545 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:17.702 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:04:18.462 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:58 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:20.035 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:20.036 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:20.036 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:20.037 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:20.038 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-02-19T22:04:20.038 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:20.038 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 5.9 M 2024-02-19T22:04:20.038 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 24 M 2024-02-19T22:04:20.038 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:20.459 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.3 MB/s | 89 kB 00:00 2024-02-19T22:04:20.652 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 110 kB/s | 21 kB 00:00 2024-02-19T22:04:20.885 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 1.1 MB/s | 578 kB 00:00 2024-02-19T22:04:21.277 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 125 kB/s | 49 kB 00:00 2024-02-19T22:04:21.711 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 336 kB/s | 355 kB 00:01 2024-02-19T22:04:22.122 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.8 MB/s | 4.8 MB 00:01 2024-02-19T22:04:22.123 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:22.123 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.8 MB/s | 5.9 MB 00:02 2024-02-19T22:04:22.176 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:22.187 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:22.187 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:22.292 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:22.295 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:22.597 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:22.874 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:04:23.193 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:04:24.176 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:04:24.542 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:04:24.753 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:04:24.999 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:04:25.274 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:25.681 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:25.833 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-02-19T22:04:26.591 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:06 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:28.163 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-02-19T22:04:28.165 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 968 k 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 3.2 M 2024-02-19T22:04:28.166 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:28.399 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 17 MB/s | 885 kB 00:00 2024-02-19T22:04:28.574 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 368 kB/s | 83 kB 00:00 2024-02-19T22:04:28.574 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:28.574 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.3 MB/s | 968 kB 00:00 2024-02-19T22:04:28.584 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:28.605 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:28.605 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:28.692 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:28.695 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:28.877 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:29.123 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:04:29.290 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:04:29.688 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:04:29.688 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:30.056 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:30.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-02-19T22:04:31.065 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:11 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:32.614 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 131 k 2024-02-19T22:04:32.616 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 439 k 2024-02-19T22:04:32.617 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:32.663 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 131 kB 00:00 2024-02-19T22:04:32.663 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:32.663 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.7 MB/s | 131 kB 00:00 2024-02-19T22:04:32.663 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:32.668 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:32.669 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:32.719 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:32.722 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:32.828 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:33.071 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:33.366 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:33.754 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:33.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-02-19T22:04:34.745 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:14 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:36.268 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:36.315 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:36.316 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:36.316 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:36.416 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-02-19T22:04:37.175 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:17 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:38.720 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:38.720 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:38.720 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:38.720 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:38.721 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 24 k 2024-02-19T22:04:38.722 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 94 k 2024-02-19T22:04:38.722 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:39.520 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 30 kB/s | 24 kB 00:00 2024-02-19T22:04:39.521 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:39.521 INFO:teuthology.orchestra.run.smithi203.stdout:Total 30 kB/s | 24 kB 00:00 2024-02-19T22:04:39.521 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:39.526 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:39.526 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:39.529 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:39.532 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:39.587 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:39.862 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:40.122 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:40.513 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:40.626 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-02-19T22:04:41.388 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:42.885 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:42.931 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:42.932 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:42.933 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:43.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-02-19T22:04:43.788 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:23 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:45.291 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:45.337 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:45.339 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:45.339 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:45.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-02-19T22:04:46.200 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:47.706 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:47.752 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:47.753 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:47.753 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:47.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-02-19T22:04:48.605 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:50.103 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:50.150 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:50.151 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:50.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-02-19T22:04:51.003 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:52.494 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:52.540 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:52.541 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:52.541 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:52.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-02-19T22:04:53.395 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:33 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:54.904 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:04:54.950 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:54.952 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-02-19T22:04:54.952 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:55.051 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-02-19T22:04:55.813 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:35 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:04:57.400 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:04:57.400 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:57.400 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:04:57.400 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:57.400 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:57.401 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 89 k 2024-02-19T22:04:57.402 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 298 k 2024-02-19T22:04:57.402 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:04:57.443 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.1 MB/s | 89 kB 00:00 2024-02-19T22:04:57.444 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:04:57.444 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.0 MB/s | 89 kB 00:00 2024-02-19T22:04:57.445 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:04:57.454 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:04:57.454 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:04:57.504 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:04:57.507 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:04:57.612 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:04:57.817 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:58.094 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:04:58.452 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:04:58.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-02-19T22:04:59.420 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:39 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:05:00.976 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:05:00.976 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:00.976 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:00.976 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:00.977 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.7 M 2024-02-19T22:05:00.978 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 18 M 2024-02-19T22:05:00.978 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:05:01.072 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 39 MB/s | 3.7 MB 00:00 2024-02-19T22:05:01.073 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:01.073 INFO:teuthology.orchestra.run.smithi203.stdout:Total 39 MB/s | 3.7 MB 00:00 2024-02-19T22:05:01.074 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:05:01.085 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:05:01.085 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:05:01.148 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:05:01.151 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:05:01.623 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:05:01.843 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:02.265 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:02.671 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:02.672 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:02.672 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:05:02.672 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:02.672 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:02.672 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:05:02.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-02-19T22:05:03.653 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:43 ago on Mon 19 Feb 2024 10:02:20 PM UTC. 2024-02-19T22:05:05.214 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-02-19T22:05:05.214 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-02-19T22:05:05.215 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:05.216 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 173 k 2024-02-19T22:05:05.216 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 649 k 2024-02-19T22:05:05.216 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-02-19T22:05:05.261 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2024-02-19T22:05:05.261 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:05:05.262 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.7 MB/s | 173 kB 00:00 2024-02-19T22:05:05.262 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-02-19T22:05:05.272 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-02-19T22:05:05.272 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-02-19T22:05:05.322 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-02-19T22:05:05.324 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-02-19T22:05:05.443 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-02-19T22:05:05.661 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:05.955 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:05:06.342 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-02-19T22:05:06.604 DEBUG:teuthology.parallel:result is None 2024-02-19T22:08:49.342 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 42 kB/s | 16 MB 06:33 2024-02-19T22:08:53.166 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 426 kB/s | 24 kB 00:00 2024-02-19T22:08:56.704 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:08:56.707 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:08:56.707 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:08:56.707 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:08:56.707 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrading: 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-02-19T22:08:56.708 INFO:teuthology.orchestra.run.smithi111.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-02-19T22:08:56.709 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:Install 25 Packages 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout:Upgrade 2 Packages 2024-02-19T22:08:56.710 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:08:56.711 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 59 M 2024-02-19T22:08:56.711 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:08:57.484 INFO:teuthology.orchestra.run.smithi111.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 23 MB/s | 6.0 MB 00:00 2024-02-19T22:08:57.509 INFO:teuthology.orchestra.run.smithi111.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 995 kB/s | 25 kB 00:00 2024-02-19T22:08:57.726 INFO:teuthology.orchestra.run.smithi111.stdout:(3/27): ceph-common-16.2.4-0.el8.x86_64.rpm 48 MB/s | 24 MB 00:00 2024-02-19T22:08:57.768 INFO:teuthology.orchestra.run.smithi111.stdout:(4/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-02-19T22:08:57.793 INFO:teuthology.orchestra.run.smithi111.stdout:(5/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 7.1 MB/s | 486 kB 00:00 2024-02-19T22:08:57.818 INFO:teuthology.orchestra.run.smithi111.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2024-02-19T22:08:57.844 INFO:teuthology.orchestra.run.smithi111.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-02-19T22:08:57.869 INFO:teuthology.orchestra.run.smithi111.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-02-19T22:08:57.903 INFO:teuthology.orchestra.run.smithi111.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-02-19T22:08:58.028 INFO:teuthology.orchestra.run.smithi111.stdout:(10/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 16 MB/s | 13 MB 00:00 2024-02-19T22:08:58.053 INFO:teuthology.orchestra.run.smithi111.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 368 kB 00:00 2024-02-19T22:08:58.079 INFO:teuthology.orchestra.run.smithi111.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2024-02-19T22:08:58.146 INFO:teuthology.orchestra.run.smithi111.stdout:(13/27): librgw2-16.2.4-0.el8.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-02-19T22:08:58.263 INFO:teuthology.orchestra.run.smithi111.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 696 kB/s | 145 kB 00:00 2024-02-19T22:08:58.330 INFO:teuthology.orchestra.run.smithi111.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 658 kB/s | 44 kB 00:00 2024-02-19T22:08:58.355 INFO:teuthology.orchestra.run.smithi111.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2024-02-19T22:08:58.397 INFO:teuthology.orchestra.run.smithi111.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 2.1 MB/s | 677 kB 00:00 2024-02-19T22:08:58.606 INFO:teuthology.orchestra.run.smithi111.stdout:(18/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 188 kB/s | 47 kB 00:00 2024-02-19T22:08:58.631 INFO:teuthology.orchestra.run.smithi111.stdout:(19/27): daxctl-libs-71.1-7.el8.x86_64.rpm 140 kB/s | 42 kB 00:00 2024-02-19T22:08:58.673 INFO:teuthology.orchestra.run.smithi111.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 291 kB/s | 80 kB 00:00 2024-02-19T22:08:58.781 INFO:teuthology.orchestra.run.smithi111.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 663 kB/s | 116 kB 00:00 2024-02-19T22:08:58.807 INFO:teuthology.orchestra.run.smithi111.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.3 MB/s | 181 kB 00:00 2024-02-19T22:08:58.890 INFO:teuthology.orchestra.run.smithi111.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 902 kB/s | 75 kB 00:00 2024-02-19T22:08:58.966 INFO:teuthology.orchestra.run.smithi111.stdout:(24/27): librados2-16.2.4-0.el8.x86_64.rpm 50 MB/s | 3.7 MB 00:00 2024-02-19T22:08:58.991 INFO:teuthology.orchestra.run.smithi111.stdout:(25/27): gperftools-libs-2.7-9.el8.x86_64.rpm 851 kB/s | 306 kB 00:00 2024-02-19T22:08:59.016 INFO:teuthology.orchestra.run.smithi111.stdout:(26/27): liboath-2.6.2-3.el8.x86_64.rpm 251 kB/s | 59 kB 00:00 2024-02-19T22:08:59.100 INFO:teuthology.orchestra.run.smithi111.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 30 MB/s | 4.0 MB 00:00 2024-02-19T22:08:59.101 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:08:59.102 INFO:teuthology.orchestra.run.smithi111.stdout:Total 25 MB/s | 59 MB 00:02 2024-02-19T22:08:59.170 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:08:59.218 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:08:59.218 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:08:59.553 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:08:59.556 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:09:00.678 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:09:00.877 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-02-19T22:09:01.122 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-02-19T22:09:01.292 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:01.697 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-02-19T22:09:01.925 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:02.026 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:02.258 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:09:02.442 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:02.504 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:02.718 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-02-19T22:09:03.276 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-02-19T22:09:03.433 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:03.511 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:03.730 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-02-19T22:09:03.883 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:03.952 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-02-19T22:09:04.139 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:09:04.317 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:09:04.491 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:04.589 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-02-19T22:09:04.741 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:05.168 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:05.366 INFO:teuthology.orchestra.run.smithi111.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:05.463 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-02-19T22:09:05.653 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-02-19T22:09:05.868 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-02-19T22:09:06.025 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:06.103 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-02-19T22:09:06.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-02-19T22:09:06.528 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-02-19T22:09:06.715 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-02-19T22:09:06.942 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-02-19T22:09:09.487 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:09.708 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:10.422 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-02-19T22:09:10.697 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:10.793 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-02-19T22:09:10.982 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:29.968 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /sys 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /proc 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /run 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /mnt 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /var/tmp 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /home 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /tmp 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout:skipping the directory /dev 2024-02-19T22:09:29.969 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:31.294 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:09:31.369 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-02-19T22:09:31.370 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:09:31.598 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-02-19T22:09:31.598 INFO:teuthology.orchestra.run.smithi111.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-02-19T22:09:32.327 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-02-19T22:09:32.328 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-02-19T22:09:32.329 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-02-19T22:09:33.322 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout:Upgraded: 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: fmt-6.2.1-1.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: leveldb-1.22-1.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.323 INFO:teuthology.orchestra.run.smithi111.stdout: liboath-2.6.2-3.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: libunwind-1.3.1-3.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.324 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-02-19T22:09:33.325 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:33.325 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:09:33.657 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-test 2024-02-19T22:09:34.410 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:41 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:09:35.957 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-02-19T22:09:35.958 INFO:teuthology.orchestra.run.smithi111.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-02-19T22:09:35.959 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:35.960 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 53 M 2024-02-19T22:09:35.960 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 229 M 2024-02-19T22:09:35.960 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:09:36.237 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 169 kB 00:00 2024-02-19T22:09:36.287 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 1.5 MB/s | 203 kB 00:00 2024-02-19T22:09:36.354 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.7 MB/s | 323 kB 00:00 2024-02-19T22:09:36.379 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-02-19T22:09:36.396 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-02-19T22:09:36.913 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 68 MB/s | 52 MB 00:00 2024-02-19T22:09:36.915 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:09:36.915 INFO:teuthology.orchestra.run.smithi111.stdout:Total 55 MB/s | 53 MB 00:00 2024-02-19T22:09:36.956 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:09:36.977 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:09:36.977 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:09:37.290 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:09:37.293 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:09:37.772 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:09:37.981 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-02-19T22:09:38.177 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-02-19T22:09:38.303 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:09:38.384 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-02-19T22:09:38.560 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-02-19T22:09:38.719 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:09:43.549 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-02-19T22:09:43.809 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-02-19T22:09:44.101 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-02-19T22:09:44.569 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:44.570 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:09:44.803 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph 2024-02-19T22:09:45.559 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:00:52 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:09:47.112 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-02-19T22:09:47.115 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-02-19T22:09:47.116 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-02-19T22:09:47.117 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout:Install 32 Packages 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:09:47.118 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 31 M 2024-02-19T22:09:47.119 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 111 M 2024-02-19T22:09:47.119 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:09:47.292 INFO:teuthology.orchestra.run.smithi111.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 104 kB/s | 6.3 kB 00:00 2024-02-19T22:09:47.342 INFO:teuthology.orchestra.run.smithi111.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-02-19T22:09:47.401 INFO:teuthology.orchestra.run.smithi111.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-02-19T22:09:47.426 INFO:teuthology.orchestra.run.smithi111.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-02-19T22:09:47.510 INFO:teuthology.orchestra.run.smithi111.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2024-02-19T22:09:47.735 INFO:teuthology.orchestra.run.smithi111.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 46 MB/s | 18 MB 00:00 2024-02-19T22:09:47.760 INFO:teuthology.orchestra.run.smithi111.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2024-02-19T22:09:47.786 INFO:teuthology.orchestra.run.smithi111.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2024-02-19T22:09:47.861 INFO:teuthology.orchestra.run.smithi111.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2024-02-19T22:09:48.028 INFO:teuthology.orchestra.run.smithi111.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 354 kB/s | 103 kB 00:00 2024-02-19T22:09:48.162 INFO:teuthology.orchestra.run.smithi111.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 829 kB/s | 249 kB 00:00 2024-02-19T22:09:48.220 INFO:teuthology.orchestra.run.smithi111.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2024-02-19T22:09:48.370 INFO:teuthology.orchestra.run.smithi111.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 291 kB/s | 44 kB 00:00 2024-02-19T22:09:48.404 INFO:teuthology.orchestra.run.smithi111.stdout:(14/32): python3-cryptography-3.2.1-7.el8.x86_6 1.7 MB/s | 641 kB 00:00 2024-02-19T22:09:48.496 INFO:teuthology.orchestra.run.smithi111.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.4 MB/s | 185 kB 00:00 2024-02-19T22:09:48.546 INFO:teuthology.orchestra.run.smithi111.stdout:(16/32): python3-werkzeug-0.12.2-4.el8.noarch.r 601 kB/s | 457 kB 00:00 2024-02-19T22:09:48.613 INFO:teuthology.orchestra.run.smithi111.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 3.2 MB/s | 384 kB 00:00 2024-02-19T22:09:48.638 INFO:teuthology.orchestra.run.smithi111.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 737 kB/s | 173 kB 00:00 2024-02-19T22:09:48.664 INFO:teuthology.orchestra.run.smithi111.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2024-02-19T22:09:48.689 INFO:teuthology.orchestra.run.smithi111.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 234 kB/s | 18 kB 00:00 2024-02-19T22:09:48.753 INFO:teuthology.orchestra.run.smithi111.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 549 kB/s | 49 kB 00:00 2024-02-19T22:09:48.781 INFO:teuthology.orchestra.run.smithi111.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 502 kB/s | 59 kB 00:00 2024-02-19T22:09:48.806 INFO:teuthology.orchestra.run.smithi111.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2024-02-19T22:09:48.832 INFO:teuthology.orchestra.run.smithi111.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 199 kB/s | 16 kB 00:00 2024-02-19T22:09:48.857 INFO:teuthology.orchestra.run.smithi111.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 248 kB/s | 19 kB 00:00 2024-02-19T22:09:48.882 INFO:teuthology.orchestra.run.smithi111.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 317 kB/s | 24 kB 00:00 2024-02-19T22:09:48.908 INFO:teuthology.orchestra.run.smithi111.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 388 kB/s | 29 kB 00:00 2024-02-19T22:09:48.933 INFO:teuthology.orchestra.run.smithi111.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 365 kB/s | 27 kB 00:00 2024-02-19T22:09:48.959 INFO:teuthology.orchestra.run.smithi111.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2024-02-19T22:09:48.992 INFO:teuthology.orchestra.run.smithi111.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2024-02-19T22:09:49.017 INFO:teuthology.orchestra.run.smithi111.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 393 kB/s | 23 kB 00:00 2024-02-19T22:09:49.043 INFO:teuthology.orchestra.run.smithi111.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 782 kB/s | 85 kB 00:00 2024-02-19T22:09:49.044 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:09:49.044 INFO:teuthology.orchestra.run.smithi111.stdout:Total 16 MB/s | 31 MB 00:01 2024-02-19T22:09:49.126 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:09:49.165 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:09:49.165 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:09:49.476 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:09:49.479 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:09:50.586 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:09:50.789 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-02-19T22:09:50.992 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-02-19T22:09:51.225 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-02-19T22:09:51.466 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-02-19T22:09:51.639 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-02-19T22:09:51.819 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-02-19T22:09:51.979 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-02-19T22:09:52.140 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-02-19T22:09:52.364 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-02-19T22:09:52.578 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-02-19T22:09:52.799 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-02-19T22:09:53.048 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-02-19T22:09:53.450 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-02-19T22:09:53.728 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-02-19T22:09:53.908 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-02-19T22:09:54.193 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-02-19T22:09:54.385 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:09:54.678 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-02-19T22:09:54.873 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-02-19T22:09:55.078 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-02-19T22:09:55.280 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-02-19T22:09:55.530 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-02-19T22:09:55.737 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-02-19T22:09:56.011 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-02-19T22:09:56.330 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-02-19T22:09:56.646 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-02-19T22:09:57.052 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-02-19T22:09:57.281 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:09:58.813 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-02-19T22:09:59.031 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:09:59.463 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-02-19T22:09:59.689 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:09:59.912 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-02-19T22:10:00.100 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:00.120 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-02-19T22:10:00.274 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:00.685 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-02-19T22:10:00.686 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-02-19T22:10:00.687 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-02-19T22:10:00.688 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-02-19T22:10:00.688 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-02-19T22:10:00.688 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-02-19T22:10:00.688 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-02-19T22:10:00.689 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-02-19T22:10:00.689 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-02-19T22:10:01.527 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-6.2-6.el8.noarch 2024-02-19T22:10:01.528 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-mako-1.0.6-14.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-portend-2.6-1.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pycparser-2.14-14.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pytz-2017.2-11.el8.noarch 2024-02-19T22:10:01.529 INFO:teuthology.orchestra.run.smithi111.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:01.530 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:01.824 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-base 2024-02-19T22:10:02.578 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:09 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:04.086 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:04.134 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:04.135 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:10:04.135 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:04.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install cephadm 2024-02-19T22:10:04.994 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:11 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:06.558 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:10:06.559 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:06.560 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 71 k 2024-02-19T22:10:06.560 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 290 k 2024-02-19T22:10:06.560 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:06.606 INFO:teuthology.orchestra.run.smithi111.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-02-19T22:10:06.607 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:06.607 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-02-19T22:10:06.607 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:06.614 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:06.614 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:06.616 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:06.619 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:06.664 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:07.167 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:07.359 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:07.583 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:08.000 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-02-19T22:10:08.000 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:08.001 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:08.001 INFO:teuthology.orchestra.run.smithi111.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-02-19T22:10:08.001 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:08.001 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:08.242 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T22:10:08.996 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:15 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:10.551 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repo Size 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:10.552 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:10:10.553 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:10.553 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 163 k 2024-02-19T22:10:10.553 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 587 k 2024-02-19T22:10:10.553 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:11.112 INFO:teuthology.orchestra.run.smithi111.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 292 kB/s | 163 kB 00:00 2024-02-19T22:10:11.113 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:11.113 INFO:teuthology.orchestra.run.smithi111.stdout:Total 291 kB/s | 163 kB 00:00 2024-02-19T22:10:11.113 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:11.125 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:11.125 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:11.177 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:11.181 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:11.503 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:11.707 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:12.132 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:12.547 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:12.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr 2024-02-19T22:10:13.517 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:20 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:10:15.049 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:15.050 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:10:15.050 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:15.160 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T22:10:15.910 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:17.465 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-02-19T22:10:17.466 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.0 M 2024-02-19T22:10:17.467 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 15 M 2024-02-19T22:10:17.468 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:17.690 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 485 kB/s | 24 kB 00:00 2024-02-19T22:10:17.708 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 134 kB/s | 8.9 kB 00:00 2024-02-19T22:10:17.766 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 21 MB/s | 2.7 MB 00:00 2024-02-19T22:10:17.850 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 237 kB/s | 34 kB 00:00 2024-02-19T22:10:17.916 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2024-02-19T22:10:17.941 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2024-02-19T22:10:17.944 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:17.945 INFO:teuthology.orchestra.run.smithi111.stdout:Total 6.2 MB/s | 3.0 MB 00:00 2024-02-19T22:10:17.954 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:17.963 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:17.963 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:18.128 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:18.131 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:18.397 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:18.640 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-02-19T22:10:18.900 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-02-19T22:10:19.099 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-02-19T22:10:19.220 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-02-19T22:10:20.355 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-02-19T22:10:20.619 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-02-19T22:10:20.907 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-02-19T22:10:21.296 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-02-19T22:10:21.296 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: python3-routes-2.4.1-12.el8.noarch 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:21.297 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:21.420 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T22:10:22.172 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:23.717 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:23.718 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-02-19T22:10:23.718 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:10:23.718 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:23.719 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout:============================================================================================ 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout:Install 8 Packages 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 37 M 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 210 M 2024-02-19T22:10:23.720 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:23.963 INFO:teuthology.orchestra.run.smithi111.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 53 MB/s | 8.6 MB 00:00 2024-02-19T22:10:25.013 INFO:teuthology.orchestra.run.smithi111.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.0 MB/s | 4.9 MB 00:01 2024-02-19T22:10:25.196 INFO:teuthology.orchestra.run.smithi111.stdout:(3/8): openblas-0.3.15-4.el8.x86_64.rpm 3.4 MB/s | 4.7 MB 00:01 2024-02-19T22:10:25.222 INFO:teuthology.orchestra.run.smithi111.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2024-02-19T22:10:25.581 INFO:teuthology.orchestra.run.smithi111.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2024-02-19T22:10:25.679 INFO:teuthology.orchestra.run.smithi111.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2024-02-19T22:10:27.867 INFO:teuthology.orchestra.run.smithi111.stdout:(7/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 979 kB/s | 3.7 MB 00:03 2024-02-19T22:10:31.625 INFO:teuthology.orchestra.run.smithi111.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 2.2 MB/s | 14 MB 00:06 2024-02-19T22:10:31.625 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:31.625 INFO:teuthology.orchestra.run.smithi111.stdout:Total 4.7 MB/s | 37 MB 00:07 2024-02-19T22:10:31.856 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:31.874 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:31.875 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:32.205 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:32.207 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:32.918 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:33.520 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:10:33.666 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-02-19T22:10:33.843 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:10:34.440 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-02-19T22:10:34.653 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:10:35.267 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:10:35.428 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:10:36.116 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-02-19T22:10:36.490 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:10:36.761 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:10:38.521 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-02-19T22:10:39.808 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-02-19T22:10:40.045 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:10:40.365 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-02-19T22:10:40.366 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-02-19T22:10:40.766 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-0.3.15-4.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:40.767 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:40.865 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-rook 2024-02-19T22:10:41.620 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:48 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:43.164 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:43.166 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:43.166 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:10:43.166 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:43.166 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-02-19T22:10:43.167 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout:Install 13 Packages 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 1.9 M 2024-02-19T22:10:43.168 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 24 M 2024-02-19T22:10:43.169 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:43.422 INFO:teuthology.orchestra.run.smithi111.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 647 kB/s | 27 kB 00:00 2024-02-19T22:10:43.490 INFO:teuthology.orchestra.run.smithi111.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 184 kB/s | 20 kB 00:00 2024-02-19T22:10:43.515 INFO:teuthology.orchestra.run.smithi111.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 199 kB/s | 27 kB 00:00 2024-02-19T22:10:43.540 INFO:teuthology.orchestra.run.smithi111.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2024-02-19T22:10:43.587 INFO:teuthology.orchestra.run.smithi111.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.1 MB/s | 110 kB 00:00 2024-02-19T22:10:43.682 INFO:teuthology.orchestra.run.smithi111.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 348 kB/s | 33 kB 00:00 2024-02-19T22:10:43.724 INFO:teuthology.orchestra.run.smithi111.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 390 kB/s | 16 kB 00:00 2024-02-19T22:10:43.749 INFO:teuthology.orchestra.run.smithi111.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 203 kB/s | 43 kB 00:00 2024-02-19T22:10:43.808 INFO:teuthology.orchestra.run.smithi111.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 973 kB/s | 81 kB 00:00 2024-02-19T22:10:43.833 INFO:teuthology.orchestra.run.smithi111.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 487 kB/s | 155 kB 00:00 2024-02-19T22:10:43.867 INFO:teuthology.orchestra.run.smithi111.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 1.1 MB/s | 64 kB 00:00 2024-02-19T22:10:43.950 INFO:teuthology.orchestra.run.smithi111.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 523 kB/s | 61 kB 00:00 2024-02-19T22:10:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.2 MB/s | 1.2 MB 00:00 2024-02-19T22:10:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:44.118 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.0 MB/s | 1.9 MB 00:00 2024-02-19T22:10:44.172 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:44.187 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:44.273 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:44.276 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:44.806 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:45.028 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-02-19T22:10:45.218 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-02-19T22:10:45.461 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-02-19T22:10:45.691 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-02-19T22:10:45.899 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-02-19T22:10:46.149 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-02-19T22:10:46.466 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-02-19T22:10:46.723 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-02-19T22:10:47.756 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-02-19T22:10:48.032 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-02-19T22:10:48.283 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-02-19T22:10:48.466 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-02-19T22:10:48.642 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-02-19T22:10:48.937 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-02-19T22:10:48.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-02-19T22:10:48.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-02-19T22:10:48.938 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-02-19T22:10:49.411 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-02-19T22:10:49.411 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-rsa-4.9-2.el8.noarch 2024-02-19T22:10:49.412 INFO:teuthology.orchestra.run.smithi111.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-02-19T22:10:49.413 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:49.413 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:49.529 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T22:10:50.280 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:01:57 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:10:51.815 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: Package Arch Version Repository Size 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2 noarch 2.10.1-4.el8 appstream 578 k 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:51.817 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout:Install 6 Packages 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 5.9 M 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 24 M 2024-02-19T22:10:51.818 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:10:52.266 INFO:teuthology.orchestra.run.smithi111.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.1 MB/s | 89 kB 00:00 2024-02-19T22:10:52.517 INFO:teuthology.orchestra.run.smithi111.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 85 kB/s | 21 kB 00:00 2024-02-19T22:10:52.959 INFO:teuthology.orchestra.run.smithi111.stdout:(3/6): python3-execnet-1.7.1-1.el8.noarch.rpm 805 kB/s | 355 kB 00:00 2024-02-19T22:10:53.101 INFO:teuthology.orchestra.run.smithi111.stdout:(4/6): python3-remoto-1.2.1-1.el8.noarch.rpm 345 kB/s | 49 kB 00:00 2024-02-19T22:10:53.476 INFO:teuthology.orchestra.run.smithi111.stdout:(5/6): python3-jinja2-2.10.1-4.el8.noarch.rpm 462 kB/s | 578 kB 00:01 2024-02-19T22:10:56.078 INFO:teuthology.orchestra.run.smithi111.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 1.2 MB/s | 4.8 MB 00:03 2024-02-19T22:10:56.079 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:10:56.079 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.4 MB/s | 5.9 MB 00:04 2024-02-19T22:10:56.133 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:10:56.143 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:10:56.143 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:10:56.248 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:10:56.251 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:10:56.527 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:10:56.799 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-02-19T22:10:57.107 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-02-19T22:10:58.039 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-02-19T22:10:58.394 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-02-19T22:10:58.620 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : python3-jinja2-2.10.1-4.el8.noarch 5/6 2024-02-19T22:10:58.849 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-jinja2-2.10.1-4.el8.noarch 3/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-02-19T22:10:59.137 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-02-19T22:10:59.808 INFO:teuthology.orchestra.run.smithi111.stdout: python3-jinja2-2.10.1-4.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-02-19T22:10:59.809 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:10:59.809 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:10:59.946 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install ceph-fuse 2024-02-19T22:11:00.707 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:07 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:02.250 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:02.251 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:02.252 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:02.252 INFO:teuthology.orchestra.run.smithi111.stdout:Install 2 Packages 2024-02-19T22:11:02.252 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:02.252 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 968 k 2024-02-19T22:11:02.252 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 3.2 M 2024-02-19T22:11:02.253 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:02.378 INFO:teuthology.orchestra.run.smithi111.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2024-02-19T22:11:02.603 INFO:teuthology.orchestra.run.smithi111.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 291 kB/s | 83 kB 00:00 2024-02-19T22:11:02.604 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:02.604 INFO:teuthology.orchestra.run.smithi111.stdout:Total 2.7 MB/s | 968 kB 00:00 2024-02-19T22:11:02.614 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:02.634 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:02.634 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:02.721 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:02.725 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:02.893 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:03.182 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-02-19T22:11:03.423 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:03.807 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-02-19T22:11:03.807 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-02-19T22:11:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-02-19T22:11:04.205 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:04.205 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-02-19T22:11:04.206 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:04.206 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:04.477 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados-devel 2024-02-19T22:11:05.242 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:12 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:06.806 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 131 k 2024-02-19T22:11:06.807 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 439 k 2024-02-19T22:11:06.808 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:06.849 INFO:teuthology.orchestra.run.smithi111.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2024-02-19T22:11:06.849 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:06.850 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.0 MB/s | 131 kB 00:00 2024-02-19T22:11:06.850 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:06.855 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:06.855 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:06.906 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:06.909 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:07.082 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:07.297 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:07.582 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:07.946 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:08.156 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs2 2024-02-19T22:11:08.924 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:15 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:10.420 INFO:teuthology.orchestra.run.smithi111.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:10.467 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:10.468 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:10.468 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:10.563 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install libcephfs-devel 2024-02-19T22:11:11.320 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:18 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:12.871 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:11:12.872 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:12.873 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 24 k 2024-02-19T22:11:12.873 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 94 k 2024-02-19T22:11:12.873 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:12.923 INFO:teuthology.orchestra.run.smithi111.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 491 kB/s | 24 kB 00:00 2024-02-19T22:11:12.923 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:12.923 INFO:teuthology.orchestra.run.smithi111.stdout:Total 479 kB/s | 24 kB 00:00 2024-02-19T22:11:12.924 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:12.927 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:12.929 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:12.932 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:12.983 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:13.181 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:13.429 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:13.769 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:13.769 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:13.770 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:13.890 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librados2 2024-02-19T22:11:14.643 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:21 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:16.164 INFO:teuthology.orchestra.run.smithi111.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:16.209 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:16.211 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:16.211 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:16.314 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install librbd1 2024-02-19T22:11:17.065 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:24 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:18.580 INFO:teuthology.orchestra.run.smithi111.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:18.627 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:18.628 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:18.628 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:18.740 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rados 2024-02-19T22:11:19.498 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:26 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:20.994 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:21.040 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:21.041 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:21.041 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:21.137 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rgw 2024-02-19T22:11:21.898 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:28 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:23.411 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:23.458 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:23.459 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:23.459 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:23.556 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-cephfs 2024-02-19T22:11:24.302 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:31 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:25.803 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:25.848 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:25.850 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:25.850 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:25.953 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install python3-rbd 2024-02-19T22:11:26.710 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:33 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:28.201 INFO:teuthology.orchestra.run.smithi111.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-02-19T22:11:28.298 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:28.298 INFO:teuthology.orchestra.run.smithi111.stdout:Nothing to do. 2024-02-19T22:11:28.298 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:28.344 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-fuse 2024-02-19T22:11:29.097 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:36 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:30.652 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:30.653 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 89 k 2024-02-19T22:11:30.654 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 298 k 2024-02-19T22:11:30.654 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:30.703 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2024-02-19T22:11:30.703 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:30.704 INFO:teuthology.orchestra.run.smithi111.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-02-19T22:11:30.704 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:30.713 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:30.713 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:30.761 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:30.764 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:30.865 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:31.068 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:31.357 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:31.727 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:31.728 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:31.728 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:31.728 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:31.728 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:31.728 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:31.913 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-mirror 2024-02-19T22:11:32.682 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:39 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:34.230 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:34.231 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:34.231 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 3.7 M 2024-02-19T22:11:34.232 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 18 M 2024-02-19T22:11:34.233 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:34.332 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 37 MB/s | 3.7 MB 00:00 2024-02-19T22:11:34.332 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:34.333 INFO:teuthology.orchestra.run.smithi111.stdout:Total 37 MB/s | 3.7 MB 00:00 2024-02-19T22:11:34.333 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:34.345 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:34.345 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:34.408 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:34.410 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:34.874 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:35.102 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:35.509 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:35.892 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:36.097 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum -y install rbd-nbd 2024-02-19T22:11:36.865 INFO:teuthology.orchestra.run.smithi111.stdout:Last metadata expiration check: 0:02:43 ago on Mon 19 Feb 2024 10:08:53 PM UTC. 2024-02-19T22:11:38.419 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-02-19T22:11:38.420 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:38.420 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Install 1 Package 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 173 k 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 649 k 2024-02-19T22:11:38.421 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-02-19T22:11:38.468 INFO:teuthology.orchestra.run.smithi111.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.7 MB/s | 173 kB 00:00 2024-02-19T22:11:38.468 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-02-19T22:11:38.469 INFO:teuthology.orchestra.run.smithi111.stdout:Total 3.6 MB/s | 173 kB 00:00 2024-02-19T22:11:38.469 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-02-19T22:11:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-02-19T22:11:38.479 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-02-19T22:11:38.530 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-02-19T22:11:38.533 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-02-19T22:11:38.650 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-02-19T22:11:38.893 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:39.193 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-02-19T22:11:39.603 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-02-19T22:11:39.821 DEBUG:teuthology.parallel:result is None 2024-02-19T22:11:39.821 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:11:39.821 INFO:teuthology.packaging:ref: None 2024-02-19T22:11:39.821 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:11:39.822 INFO:teuthology.packaging:branch: None 2024-02-19T22:11:39.822 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:11:39.822 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:11:40.052 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:11:40.075 INFO:teuthology.orchestra.run.smithi111.stdout:16.2.4-0.el8 2024-02-19T22:11:40.075 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:11:40.075 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:11:40.077 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-02-19T22:11:40.077 INFO:teuthology.packaging:ref: None 2024-02-19T22:11:40.077 INFO:teuthology.packaging:tag: v16.2.4 2024-02-19T22:11:40.078 INFO:teuthology.packaging:branch: None 2024-02-19T22:11:40.078 INFO:teuthology.packaging:sha1: 81bd20d634209c7cb82c18be12b4b5a05643ebf1 2024-02-19T22:11:40.078 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-02-19T22:11:40.233 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T22:11:40.263 INFO:teuthology.orchestra.run.smithi203.stdout:16.2.4-0.el8 2024-02-19T22:11:40.263 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-02-19T22:11:40.264 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-02-19T22:11:40.265 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T22:11:40.266 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:11:40.266 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:11:40.293 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:11:40.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T22:11:40.338 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T22:11:40.339 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:11:40.339 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:11:40.365 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:11:40.432 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:11:40.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T22:11:40.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T22:11:40.532 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T22:11:40.532 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:11:40.532 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:11:40.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:11:40.625 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:11:40.625 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T22:11:40.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T22:11:40.722 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T22:11:40.723 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:11:40.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:11:40.748 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:11:40.815 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:11:40.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T22:11:40.844 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T22:11:40.911 INFO:teuthology.run_tasks:Running task print... 2024-02-19T22:11:40.921 INFO:teuthology.task.print:**** done install task... 2024-02-19T22:11:40.921 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T22:11:40.992 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(PG_DEGRADED\\)', 'Degraded data redundancy', '\\(PG_', 'MDS_INSUFFICIENT_STANDBY', 'deprecated feature inline_data', 'compat changed unexpectedly', 'overall HEALTH_', 'OSD_DOWN', 'OSD_', 'but it is still running', 'is not responding'], 'sha1': '81bd20d634209c7cb82c18be12b4b5a05643ebf1'} 2024-02-19T22:11:40.992 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-02-19T22:11:40.993 INFO:tasks.cephadm:Cluster fsid is dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:11:40.993 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T22:11:40.993 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T22:11:40.993 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi111': '172.21.15.111', 'mon.smithi203': '172.21.15.203'} 2024-02-19T22:11:40.993 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T22:11:40.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-02-19T22:11:41.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo hostname $(hostname -s) 2024-02-19T22:11:41.048 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-02-19T22:11:41.049 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.197 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:11 /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.198 DEBUG:teuthology.orchestra.run.smithi203:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.333 INFO:teuthology.orchestra.run.smithi203.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Feb 19 22:11 /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.333 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.351 DEBUG:teuthology.orchestra.run.smithi203:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T22:11:41.381 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-02-19T22:11:41.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph 2024-02-19T22:11:41.447 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-02-19T22:11:41.485 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /etc/ceph 2024-02-19T22:11:41.513 INFO:tasks.cephadm:Writing seed config... 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T22:11:41.515 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T22:11:41.516 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T22:11:41.517 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T22:11:41.518 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:11:41.518 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T22:11:41.543 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = dbbe4146-cf73-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T22:11:41.543 DEBUG:teuthology.orchestra.run.smithi111:mon.smithi111> sudo journalctl -f -n 0 -u ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service 2024-02-19T22:11:41.585 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T22:11:41.586 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.111 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:11:41.611 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:-- Logs begin at Mon 2024-02-19 21:54:51 UTC. -- 2024-02-19T22:11:41.848 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:11:42.818 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:11:42.833 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying podman|docker is present... 2024-02-19T22:11:42.833 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:11:42.857 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:11:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying lvm2 is present... 2024-02-19T22:11:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying time synchronization is in place... 2024-02-19T22:11:42.874 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:11:42.881 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:11:42.882 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:11:42.887 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:11:42.888 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:11:42.894 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: enabled 2024-02-19T22:11:42.895 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:11:42.900 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: active 2024-02-19T22:11:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:11:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:Repeating the final host check... 2024-02-19T22:11:42.901 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-02-19T22:11:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.3-dev 2024-02-19T22:11:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:podman|docker (/bin/podman) is present 2024-02-19T22:11:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl is present 2024-02-19T22:11:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:lvcreate is present 2024-02-19T22:11:42.932 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled chrony.service 2024-02-19T22:11:42.938 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T22:11:42.939 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active chrony.service 2024-02-19T22:11:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:11:42.944 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled chronyd.service 2024-02-19T22:11:42.949 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: enabled 2024-02-19T22:11:42.950 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active chronyd.service 2024-02-19T22:11:42.955 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: active 2024-02-19T22:11:42.956 INFO:teuthology.orchestra.run.smithi111.stderr:Unit chronyd.service is enabled and running 2024-02-19T22:11:42.956 INFO:teuthology.orchestra.run.smithi111.stderr:Host looks OK 2024-02-19T22:11:42.956 INFO:teuthology.orchestra.run.smithi111.stderr:Cluster fsid: dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:11:42.956 INFO:teuthology.orchestra.run.smithi111.stderr:Acquiring lock 140198187514344 on /run/cephadm/dbbe4146-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:11:42.956 INFO:teuthology.orchestra.run.smithi111.stderr:Lock 140198187514344 acquired on /run/cephadm/dbbe4146-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:11:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying IP 172.21.15.111 port 3300 ... 2024-02-19T22:11:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying IP 172.21.15.111 port 6789 ... 2024-02-19T22:11:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:Base mon IP is 172.21.15.111, final addrv is [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-02-19T22:11:42.957 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /sbin/ip route ls 2024-02-19T22:11:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.111 metric 100 2024-02-19T22:11:42.959 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.111 metric 100 2024-02-19T22:11:42.960 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /sbin/ip -6 route ls 2024-02-19T22:11:42.962 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T22:11:42.962 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T22:11:42.962 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-02-19T22:11:42.963 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /sbin/ip -6 addr ls 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2024-02-19T22:11:42.965 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-02-19T22:11:42.966 INFO:teuthology.orchestra.run.smithi111.stderr:Mon IP 172.21.15.111 is in CIDR network 172.21.0.0/20 2024-02-19T22:11:42.966 INFO:teuthology.orchestra.run.smithi111.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T22:11:42.967 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:11:42.968 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-02-19T22:11:43.055 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-02-19T22:11:49.577 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Getting image source signatures 2024-02-19T22:11:49.583 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-02-19T22:11:49.583 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-02-19T22:12:01.748 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:12:01.907 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Writing manifest to image destination 2024-02-19T22:12:02.207 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-02-19T22:12:02.218 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-02-19T22:12:02.847 INFO:teuthology.orchestra.run.smithi111.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:12:03.511 INFO:teuthology.orchestra.run.smithi111.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-02-19T22:12:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:Extracting ceph user uid/gid from container image... 2024-02-19T22:12:03.512 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-02-19T22:12:04.027 INFO:teuthology.orchestra.run.smithi111.stderr:stat: 167 167 2024-02-19T22:12:04.556 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial keys... 2024-02-19T22:12:04.557 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:05.083 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQA00tNloPJHOxAArQjS0SWoPD1SIAFUff/i/w== 2024-02-19T22:12:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:06.114 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQA20tNluXVRARAA/zZ5L9sJCh94JZDyTzPtuQ== 2024-02-19T22:12:06.659 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-02-19T22:12:07.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQA30tNlEupGCBAABhZXUKgIuausofVoTl6NEg== 2024-02-19T22:12:07.815 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial monmap... 2024-02-19T22:12:07.815 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp2dk9ioq6:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 --addv smithi111 [v2:172.21.15.111:3300,v1:172.21.15.111:6789] /tmp/monmap 2024-02-19T22:12:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:12:08.342 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:08.343 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr:monmaptool for smithi111 [v2:172.21.15.111:3300,v1:172.21.15.111:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: set fsid to dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr:Creating mon... 2024-02-19T22:12:09.241 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/mon.smithi111:/var/lib/ceph/mon/ceph-smithi111:z -v /tmp/ceph-tmpibzqhai6:/tmp/keyring:z -v /tmp/ceph-tmp2dk9ioq6:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi111 --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.684+0000 7f3307541700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.685+0000 7f3307541700 1 imported monmap: 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: epoch 0 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: fsid dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: last_changed 2024-02-19T22:12:08.292237+0000 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: created 2024-02-19T22:12:08.292237+0000 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-02-19T22:12:09.833 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon.smithi111 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.685+0000 7f3307541700 0 /usr/bin/ceph-mon: set fsid to dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: DB SUMMARY 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 0, files: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.834 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.create_if_missing: 1 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.env: 0x557f7ea491c0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.info_log: 0x557f80f44ec0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:12:09.835 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi111/store.db 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:12:09.836 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.write_buffer_manager: 0x557f80f4b530 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:12:09.837 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.row_cache: None 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:12:09.838 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:12:09.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:12:09.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.686+0000 7f3307541700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000001 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.merge_operator: 2024-02-19T22:12:09.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557f80e5dcb8) 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: index_type: 0 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: checksum: 1 2024-02-19T22:12:09.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_cache: 0x557f80e92d10 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-02-19T22:12:09.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: format_version: 2 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: block_align: 0 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:12:09.844 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:12:09.845 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:12:09.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:12:09.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:12:09.848 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:12:09.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.687+0000 7f3307541700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.688+0000 7f3307541700 4 rocksdb: DB pointer 0x557f80f57800 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.688+0000 7f32f03da700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.688+0000 7f32f03da700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:09.852 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:09.853 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:09.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:09.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.689+0000 7f3307541700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.689+0000 7f3307541700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-02-19T22:12:09.856 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-mon: debug 2024-02-19T22:12:09.689+0000 7f3307541700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi111 for mon.smithi111 2024-02-19T22:12:10.352 INFO:teuthology.orchestra.run.smithi111.stderr:create mon.smithi111 on 2024-02-19T22:12:10.353 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:10.358 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl disable ceph.target 2024-02-19T22:12:10.365 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-02-19T22:12:10.448 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl enable ceph.target 2024-02-19T22:12:10.456 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T22:12:10.535 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl start ceph.target 2024-02-19T22:12:10.542 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl enable ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:12:10.547 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:12:10.548 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target. 2024-02-19T22:12:10.626 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target 2024-02-19T22:12:10.633 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl daemon-reload 2024-02-19T22:12:10.718 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:12:10.741 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl reset-failed ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:12:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to reset failed state of unit ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service: Unit ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service not loaded. 2024-02-19T22:12:10.747 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl enable ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:12:10.753 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target.wants/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service → /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:12:10.841 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:12:11.128 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:10 smithi111 systemd[1]: Starting Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:12:11.334 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:12:11.343 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-02-19T22:12:11.344 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:12:11.350 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:12:11.351 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-02-19T22:12:11.351 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:12:11.351 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:12:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mon to start... 2024-02-19T22:12:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mon... 2024-02-19T22:12:11.352 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/mon.smithi111:/var/lib/ceph/mon/ceph-smithi111:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-02-19T22:12:11.558 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.277618+0000 mon.smithi111 (mon.0) 0 : [INF] mkfs dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:11.558 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.271983+0000 mon.smithi111 (mon.0) 1 : cluster [INF] mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-02-19T22:12:11.559 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:11 smithi111 bash[37764]: 8337c2a5c324d97b76d59a4165dbf05b0499583101c5a77beb010595dd6c372c 2024-02-19T22:12:11.559 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:11 smithi111 systemd[1]: Started Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:12:11.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.855+0000 7f4114a3d700 1 Processor -- start 2024-02-19T22:12:11.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.855+0000 7f4114a3d700 1 -- start start 2024-02-19T22:12:11.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f4114a3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f4114a3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100fef40 con 0x7f4110070430 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41280/0 (socket says 172.21.15.111:41280) 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410e59c700 1 -- 172.21.15.111:0/259512843 learned_addr learned my addr 172.21.15.111:0/259512843 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410e59c700 1 -- 172.21.15.111:0/259512843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41100ff080 con 0x7f4110070430 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410e59c700 1 --2- 172.21.15.111:0/259512843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f410400b5f0 tx=0x7f4104006ee0).ready entity=mon.0 client_cookie=b6fd91d5a6ef72c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.856+0000 7f410d59a700 1 -- 172.21.15.111:0/259512843 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f410400d600 con 0x7f4110070430 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f410d59a700 1 -- 172.21.15.111:0/259512843 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410400d760 con 0x7f4110070430 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f410d59a700 1 -- 172.21.15.111:0/259512843 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f410400da30 con 0x7f4110070430 2024-02-19T22:12:11.862 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 -- 172.21.15.111:0/259512843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 msgr2=0x7f41100fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 --2- 172.21.15.111:0/259512843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f410400b5f0 tx=0x7f4104006ee0).stop 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 -- 172.21.15.111:0/259512843 shutdown_connections 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 --2- 172.21.15.111:0/259512843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f41100fea50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 -- 172.21.15.111:0/259512843 >> 172.21.15.111:0/259512843 conn(0x7f41100f6530 msgr2=0x7f41100f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 -- 172.21.15.111:0/259512843 shutdown_connections 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.857+0000 7f4114a3d700 1 -- 172.21.15.111:0/259512843 wait complete. 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f4114a3d700 1 Processor -- start 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f4114a3d700 1 -- start start 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f4114a3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f4114a3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411010a850 con 0x7f4110070430 2024-02-19T22:12:11.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f410e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f410e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41290/0 (socket says 172.21.15.111:41290) 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f410e59c700 1 -- 172.21.15.111:0/3324983678 learned_addr learned my addr 172.21.15.111:0/3324983678 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f410e59c700 1 -- 172.21.15.111:0/3324983678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410400b200 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.858+0000 7f410e59c700 1 --2- 172.21.15.111:0/3324983678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4104006c70 tx=0x7f4104016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4104011b40 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104011ca0 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f410401d430 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411010aa50 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f411010aee0 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f410400f070 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.859+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f4104027460 con 0x7f4110070430 2024-02-19T22:12:11.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.860+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110045ad0 con 0x7f4110070430 2024-02-19T22:12:11.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.862+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f4104016070 con 0x7f4110070430 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.929+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4110103f90 con 0x7f4110070430 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.929+0000 7f40feffd700 1 -- 172.21.15.111:0/3324983678 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f4104036070 con 0x7f4110070430 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: cluster: 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: id: dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: services: 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi111 (age 0.654573s) 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mgr: no daemons active 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.933 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: data: 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: pgs: 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 msgr2=0x7f411010a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 --2- 172.21.15.111:0/3324983678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f4104006c70 tx=0x7f4104016000).stop 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 shutdown_connections 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 --2- 172.21.15.111:0/3324983678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4110070430 0x7f411010a360 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 >> 172.21.15.111:0/3324983678 conn(0x7f41100f6530 msgr2=0x7f41100f81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 shutdown_connections 2024-02-19T22:12:11.934 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:11.930+0000 7f4114a3d700 1 -- 172.21.15.111:0/3324983678 wait complete. 2024-02-19T22:12:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:mon is available 2024-02-19T22:12:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:Assimilating anything we can from ceph.conf... 2024-02-19T22:12:12.573 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z -v /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/mon.smithi111:/var/lib/ceph/mon/ceph-smithi111:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi111/config 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.276200+0000 mon.smithi111 (mon.0) 2 : cluster [INF] mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.276675+0000 mon.smithi111 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.279463+0000 mon.smithi111 (mon.0) 4 : cluster [DBG] fsmap 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.282004+0000 mon.smithi111 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: cluster 2024-02-19T22:12:11.282383+0000 mon.smithi111 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:12:12.575 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: audit 2024-02-19T22:12:11.930743+0000 mon.smithi111 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.111:0/3324983678' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.085+0000 7f9dad922700 1 Processor -- start 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.085+0000 7f9dad922700 1 -- start start 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9dad922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9dad922700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da8072940 con 0x7f9da8072520 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41296/0 (socket says 172.21.15.111:41296) 2024-02-19T22:12:13.090 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da6ffd700 1 -- 172.21.15.111:0/854616929 learned_addr learned my addr 172.21.15.111:0/854616929 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da6ffd700 1 -- 172.21.15.111:0/854616929 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8070fb0 con 0x7f9da8072520 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da6ffd700 1 --2- 172.21.15.111:0/854616929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9d9800b5f0 tx=0x7f9d98006ee0).ready entity=mon.0 client_cookie=1b51c03fbb02b02d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.086+0000 7f9da5ffb700 1 -- 172.21.15.111:0/854616929 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9800d600 con 0x7f9da8072520 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9da5ffb700 1 -- 172.21.15.111:0/854616929 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d9800d760 con 0x7f9da8072520 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9da5ffb700 1 -- 172.21.15.111:0/854616929 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9800da30 con 0x7f9da8072520 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 -- 172.21.15.111:0/854616929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 msgr2=0x7f9da8070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 --2- 172.21.15.111:0/854616929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9d9800b5f0 tx=0x7f9d98006ee0).stop 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 -- 172.21.15.111:0/854616929 shutdown_connections 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 --2- 172.21.15.111:0/854616929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da8070ac0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 -- 172.21.15.111:0/854616929 >> 172.21.15.111:0/854616929 conn(0x7f9da80f76a0 msgr2=0x7f9da80f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 -- 172.21.15.111:0/854616929 shutdown_connections 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.087+0000 7f9dad922700 1 -- 172.21.15.111:0/854616929 wait complete. 2024-02-19T22:12:13.091 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9dad922700 1 Processor -- start 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9dad922700 1 -- start start 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9dad922700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9dad922700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da810ca80 con 0x7f9da8072520 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9da6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9da6ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41302/0 (socket says 172.21.15.111:41302) 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9da6ffd700 1 -- 172.21.15.111:0/3128331575 learned_addr learned my addr 172.21.15.111:0/3128331575 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9da6ffd700 1 -- 172.21.15.111:0/3128331575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9800b200 con 0x7f9da8072520 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.088+0000 7f9da6ffd700 1 --2- 172.21.15.111:0/3128331575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9d98006c70 tx=0x7f9d98016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:13.092 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d98011b40 con 0x7f9da8072520 2024-02-19T22:12:13.093 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d98011ca0 con 0x7f9da8072520 2024-02-19T22:12:13.093 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da810cc80 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9801d430 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da810d110 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9d9800f070 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.089+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9d98027460 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.090+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8045ad0 con 0x7f9da8072520 2024-02-19T22:12:13.094 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.092+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9d98016070 con 0x7f9da8072520 2024-02-19T22:12:13.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.156+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9da810e620 con 0x7f9da8072520 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.159+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 7 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9d9801d5e0 con 0x7f9da8072520 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.159+0000 7f9d9f7fe700 1 -- 172.21.15.111:0/3128331575 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+491 (secure 0 0 0) 0x7f9d98036020 con 0x7f9da8072520 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [global] 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: fsid = dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: log_to_journald = false 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-02-19T22:12:13.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [mgr] 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [osd] 2024-02-19T22:12:13.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 msgr2=0x7f9da810c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 --2- 172.21.15.111:0/3128331575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9d98006c70 tx=0x7f9d98016000).stop 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 shutdown_connections 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 --2- 172.21.15.111:0/3128331575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9da8072520 0x7f9da810c590 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 >> 172.21.15.111:0/3128331575 conn(0x7f9da80f76a0 msgr2=0x7f9da8100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 shutdown_connections 2024-02-19T22:12:13.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:13.160+0000 7f9dad922700 1 -- 172.21.15.111:0/3128331575 wait complete. 2024-02-19T22:12:13.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:13 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: audit 2024-02-19T22:12:13.157803+0000 mon.smithi111 (mon.0) 8 : audit [INF] from='client.? 172.21.15.111:0/3128331575' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T22:12:13.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:13 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: audit 2024-02-19T22:12:13.160773+0000 mon.smithi111 (mon.0) 9 : audit [INF] from='client.? 172.21.15.111:0/3128331575' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T22:12:13.713 INFO:teuthology.orchestra.run.smithi111.stderr:Generating new minimal ceph.conf... 2024-02-19T22:12:13.713 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z -v /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/mon.smithi111:/var/lib/ceph/mon/ceph-smithi111:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi111/config 2024-02-19T22:12:14.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.217+0000 7f77fed7d700 1 Processor -- start 2024-02-19T22:12:14.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.217+0000 7f77fed7d700 1 -- start start 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.217+0000 7f77fed7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.217+0000 7f77fed7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80fb6a0 con 0x7f77f80fad90 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77fcb19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77fcb19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41308/0 (socket says 172.21.15.111:41308) 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77fcb19700 1 -- 172.21.15.111:0/39138202 learned_addr learned my addr 172.21.15.111:0/39138202 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77fcb19700 1 -- 172.21.15.111:0/39138202 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f80fb7e0 con 0x7f77f80fad90 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77fcb19700 1 --2- 172.21.15.111:0/39138202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f77ec00b5f0 tx=0x7f77ec006ee0).ready entity=mon.0 client_cookie=5dd28c075c85b781 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77f77fe700 1 -- 172.21.15.111:0/39138202 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77ec00d600 con 0x7f77f80fad90 2024-02-19T22:12:14.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77f77fe700 1 -- 172.21.15.111:0/39138202 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f77ec00dbf0 con 0x7f77f80fad90 2024-02-19T22:12:14.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.218+0000 7f77f77fe700 1 -- 172.21.15.111:0/39138202 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77ec011cb0 con 0x7f77f80fad90 2024-02-19T22:12:14.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- 172.21.15.111:0/39138202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 msgr2=0x7f77f80fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:14.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 --2- 172.21.15.111:0/39138202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f77ec00b5f0 tx=0x7f77ec006ee0).stop 2024-02-19T22:12:14.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- 172.21.15.111:0/39138202 shutdown_connections 2024-02-19T22:12:14.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 --2- 172.21.15.111:0/39138202 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f80fb1b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- 172.21.15.111:0/39138202 >> 172.21.15.111:0/39138202 conn(0x7f77f80f6530 msgr2=0x7f77f80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- 172.21.15.111:0/39138202 shutdown_connections 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- 172.21.15.111:0/39138202 wait complete. 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 Processor -- start 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.219+0000 7f77fed7d700 1 -- start start 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fed7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fed7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f80ff4c0 con 0x7f77f80fad90 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fcb19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fcb19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41318/0 (socket says 172.21.15.111:41318) 2024-02-19T22:12:14.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fcb19700 1 -- 172.21.15.111:0/4100566571 learned_addr learned my addr 172.21.15.111:0/4100566571 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fcb19700 1 -- 172.21.15.111:0/4100566571 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec00b200 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fcb19700 1 --2- 172.21.15.111:0/4100566571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f77ec00bf40 tx=0x7f77ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77ec01b570 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f77ec01bb60 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f80ff660 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77ec01aa20 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.220+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f80ffaf0 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.221+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f77ec00f070 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.221+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f77ec025e20 con 0x7f77f80fad90 2024-02-19T22:12:14.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.222+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f8045ad0 con 0x7f77f80fad90 2024-02-19T22:12:14.226 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.224+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f77ec016070 con 0x7f77f80fad90 2024-02-19T22:12:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.288+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f77f8104c00 con 0x7f77f80fad90 2024-02-19T22:12:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.288+0000 7f77f57fa700 1 -- 172.21.15.111:0/4100566571 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f77ec025460 con 0x7f77f80fad90 2024-02-19T22:12:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 msgr2=0x7f77f8101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 --2- 172.21.15.111:0/4100566571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f77ec00bf40 tx=0x7f77ec006ee0).stop 2024-02-19T22:12:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 shutdown_connections 2024-02-19T22:12:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 --2- 172.21.15.111:0/4100566571 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f77f80fad90 0x7f77f8101040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:14.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 >> 172.21.15.111:0/4100566571 conn(0x7f77f80f6530 msgr2=0x7f77f8070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:14.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 shutdown_connections 2024-02-19T22:12:14.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:14.289+0000 7f77fed7d700 1 -- 172.21.15.111:0/4100566571 wait complete. 2024-02-19T22:12:14.641 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: audit 2024-02-19T22:12:14.289995+0000 mon.smithi111 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.111:0/4100566571' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:12:14.894 INFO:teuthology.orchestra.run.smithi111.stderr:Restarting the monitor... 2024-02-19T22:12:14.894 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl restart ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:12:15.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:14 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:12:15.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: debug 2024-02-19T22:12:15.004+0000 7f8e68892700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:12:15.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[37779]: debug 2024-02-19T22:12:15.004+0000 7f8e68892700 -1 mon.smithi111@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:12:15.647 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 podman[38106]: ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111 2024-02-19T22:12:15.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 systemd[1]: ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service: Succeeded. 2024-02-19T22:12:15.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 systemd[1]: Stopped Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:12:15.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:15 smithi111 systemd[1]: Starting Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:12:16.202 INFO:teuthology.orchestra.run.smithi111.stderr:Setting mon public_network to 172.21.0.0/20 2024-02-19T22:12:16.202 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-02-19T22:12:16.417 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.129+0000 7fb6e6397700 0 set uid:gid to 167:167 (ceph:ceph) 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.129+0000 7fb6e6397700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.129+0000 7fb6e6397700 0 pidfile_write: ignore empty --pid-file 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 0 load: jerasure load: lrc load: isa 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: RocksDB version: 6.8.1 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Compile date May 13 2021 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: DB SUMMARY 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: CURRENT file: CURRENT 2024-02-19T22:12:16.418 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: IDENTITY file: IDENTITY 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 1, files: 000004.sst 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 000006.log size: 98740 ; 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.error_if_exists: 0 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.create_if_missing: 0 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.paranoid_checks: 1 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.env: 0x5650275821c0 2024-02-19T22:12:16.419 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.fs: Posix File System 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.info_log: 0x565029389960 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_file_opening_threads: 16 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.statistics: (nil) 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.use_fsync: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_log_file_size: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.keep_log_file_num: 1000 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.recycle_log_file_num: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.allow_fallocate: 1 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.allow_mmap_reads: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.allow_mmap_writes: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.use_direct_reads: 0 2024-02-19T22:12:16.420 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.create_missing_column_families: 0 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.db_log_dir: 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi111/store.db 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_subcompactions: 1 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_background_flushes: -1 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.advise_random_on_open: 1 2024-02-19T22:12:16.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.db_write_buffer_size: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.write_buffer_manager: 0x56502938ffb0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.rate_limiter: (nil) 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.wal_recovery_mode: 2 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.enable_thread_tracking: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.enable_pipelined_write: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.unordered_write: 0 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T22:12:16.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.row_cache: None 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.wal_filter: None 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.allow_ingest_behind: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.preserve_deletes: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.two_write_queues: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.manual_wal_flush: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.atomic_flush: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.log_readahead_size: 0 2024-02-19T22:12:16.423 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_background_jobs: 2 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_background_compactions: -1 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_total_wal_size: 0 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T22:12:16.424 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.max_open_files: -1 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.bytes_per_sync: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.compaction_readahead_size: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Compression algorithms supported: 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kZSTD supported: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kXpressCompression supported: 0 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kLZ4HCCompression supported: 1 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kLZ4Compression supported: 1 2024-02-19T22:12:16.425 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kBZip2Compression supported: 0 2024-02-19T22:12:16.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kZlibCompression supported: 1 2024-02-19T22:12:16.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: kSnappyCompression supported: 1 2024-02-19T22:12:16.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T22:12:16.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000005 2024-02-19T22:12:16.427 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.134+0000 7fb6e6397700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.merge_operator: 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_filter: None 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_filter_factory: None 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5650292a1cb0) 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cache_index_and_filter_blocks: 1 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T22:12:16.428 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: pin_top_level_index_and_filter: 1 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: index_type: 0 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: data_block_index_type: 0 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: index_shortening: 1 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: data_block_hash_table_util_ratio: 0.750000 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: hash_index_allow_collision: 1 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: checksum: 1 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: no_block_cache: 0 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_cache: 0x5650292d6f10 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_cache_name: BinnedLRUCache 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_cache_options: 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: capacity : 536870912 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: num_shard_bits : 4 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: strict_capacity_limit : 0 2024-02-19T22:12:16.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: high_pri_pool_ratio: 0.000 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_cache_compressed: (nil) 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: persistent_cache: (nil) 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_size: 4096 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_size_deviation: 10 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_restart_interval: 16 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: index_block_restart_interval: 1 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: metadata_block_size: 4096 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: partition_filters: 0 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: use_delta_encoding: 1 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: whole_key_filtering: 1 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: verify_compression: 0 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: read_amp_bytes_per_bit: 0 2024-02-19T22:12:16.430 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: format_version: 2 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: enable_index_compression: 1 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: block_align: 0 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.write_buffer_size: 33554432 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_write_buffer_number: 2 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression: NoCompression 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression: Disabled 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.prefix_extractor: nullptr 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.num_levels: 7 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T22:12:16.431 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.level: 32767 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.strategy: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compression_opts.enabled: false 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T22:12:16.432 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.target_file_size_base: 67108864 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T22:12:16.433 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.arena_block_size: 4194304 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.disable_auto_compactions: 0 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T22:12:16.434 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.table_properties_collectors: 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.inplace_update_support: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.bloom_locality: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.max_successive_merges: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.paranoid_file_checks: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.force_consistency_checks: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.report_bg_io_stats: 0 2024-02-19T22:12:16.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.ttl: 2592000 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380736136998, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.135+0000 7fb6e6397700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.136+0000 7fb6e6397700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.136+0000 7fb6e6397700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380736137845, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 94314, "table_properties": {"data_size": 92402, "index_size": 389, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12563, "raw_average_key_size": 49, "raw_value_size": 85433, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 254, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708380736, "oldest_key_time": 3, "file_creation_time": 0}} 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.136+0000 7fb6e6397700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.136+0000 7fb6e6397700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1708380736138237, "job": 1, "event": "recovery_finished"} 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.137+0000 7fb6e6397700 4 rocksdb: DB pointer 0x56502939b800 2024-02-19T22:12:16.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.137+0000 7fb6cd1af700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.137+0000 7fb6cd1af700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** DB Stats ** 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** Compaction Stats [default] ** 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:16.437 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** Compaction Stats [default] ** 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:16.438 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Cumulative compaction: 0.00 GB write, 34.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Interval compaction: 0.00 GB write, 34.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** Compaction Stats [default] ** 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: L0 2/0 93.80 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Sum 2/0 93.80 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.439 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** Compaction Stats [default] ** 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.3 0.00 0.00 1 0.001 0 0 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: AddFile(Keys): cumulative 0, interval 0 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Cumulative compaction: 0.00 GB write, 33.88 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T22:12:16.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ** File Read Latency Histogram By Level [default] ** 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.138+0000 7fb6e6397700 0 starting mon.smithi111 rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi111 fsid dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 1 mon.smithi111@-1(???) e1 preinit fsid dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).mds e1 new map 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).mds e1 print_map 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: e1 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: legacy client fscid: -1 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: No filesystems configured 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: 2024-02-19T22:12:16.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 0 mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:14.289995+0000 mon.smithi111 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.111:0/4100566571' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.139+0000 7fb6e6397700 1 mon.smithi111@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expand map: {default=false} 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta from 'false' to 'false' 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expanded map: {default=false} 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expand map: {default=info} 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta from 'info' to 'info' 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expanded map: {default=info} 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expand map: {default=daemon} 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-02-19T22:12:16.442 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expanded map: {default=daemon} 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expand map: {default=debug} 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta from 'debug' to 'debug' 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:12:16.140+0000 7fb6e6397700 20 expand_channel_meta expanded map: {default=debug} 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 bash[38180]: ef54b9647109c05f5c7dce41ed62145a45a15b0482b4134544ea46bc6a3f554d 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 systemd[1]: Started Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715. 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:16.157089+0000 mon.smithi111 (mon.0) 1 : cluster [INF] mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:16.157146+0000 mon.smithi111 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:16.157207+0000 mon.smithi111 (mon.0) 3 : cluster [DBG] fsmap 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:16.157244+0000 mon.smithi111 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-02-19T22:12:16.443 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:16.157662+0000 mon.smithi111 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.701+0000 7f8329644700 1 Processor -- start 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.701+0000 7f8329644700 1 -- start start 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.701+0000 7f8329644700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.701+0000 7f8329644700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83240fb6a0 con 0x7f83240fad90 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8322ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:16.706 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8322ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41320/0 (socket says 172.21.15.111:41320) 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8322ffd700 1 -- 172.21.15.111:0/2541698780 learned_addr learned my addr 172.21.15.111:0/2541698780 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8322ffd700 1 -- 172.21.15.111:0/2541698780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83240fb7e0 con 0x7f83240fad90 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8322ffd700 1 --2- 172.21.15.111:0/2541698780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f831800b870 tx=0x7f8318006f70).ready entity=mon.0 client_cookie=4cb3455ae850f294 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8321ffb700 1 -- 172.21.15.111:0/2541698780 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f831800da20 con 0x7f83240fad90 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8321ffb700 1 -- 172.21.15.111:0/2541698780 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f8318010470 con 0x7f83240fad90 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.702+0000 7f8321ffb700 1 -- 172.21.15.111:0/2541698780 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f831800f5b0 con 0x7f83240fad90 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 -- 172.21.15.111:0/2541698780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 msgr2=0x7f83240fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 --2- 172.21.15.111:0/2541698780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f831800b870 tx=0x7f8318006f70).stop 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 -- 172.21.15.111:0/2541698780 shutdown_connections 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 --2- 172.21.15.111:0/2541698780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f83240fb1b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 -- 172.21.15.111:0/2541698780 >> 172.21.15.111:0/2541698780 conn(0x7f83240f6530 msgr2=0x7f83240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 -- 172.21.15.111:0/2541698780 shutdown_connections 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.703+0000 7f8329644700 1 -- 172.21.15.111:0/2541698780 wait complete. 2024-02-19T22:12:16.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8329644700 1 Processor -- start 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8329644700 1 -- start start 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8329644700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8329644700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324108df0 con 0x7f83240fad90 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8322ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8322ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41334/0 (socket says 172.21.15.111:41334) 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8322ffd700 1 -- 172.21.15.111:0/2256985876 learned_addr learned my addr 172.21.15.111:0/2256985876 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8322ffd700 1 -- 172.21.15.111:0/2256985876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831800b480 con 0x7f83240fad90 2024-02-19T22:12:16.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.704+0000 7f8322ffd700 1 --2- 172.21.15.111:0/2256985876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8318006800 tx=0x7f8318006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f831800f3d0 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f831800fd00 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8324108ff0 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8318019d00 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8324109480 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f8318016030 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.705+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f831802d3f0 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.706+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324045ad0 con 0x7f83240fad90 2024-02-19T22:12:16.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.708+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f8318014070 con 0x7f83240fad90 2024-02-19T22:12:16.774 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.772+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f83240fcae0 con 0x7f83240fad90 2024-02-19T22:12:16.776 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.774+0000 7f830b7fe700 1 -- 172.21.15.111:0/2256985876 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f831800fe70 con 0x7f83240fad90 2024-02-19T22:12:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 msgr2=0x7f8324108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 --2- 172.21.15.111:0/2256985876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f8318006800 tx=0x7f8318006f70).stop 2024-02-19T22:12:16.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 shutdown_connections 2024-02-19T22:12:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 --2- 172.21.15.111:0/2256985876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83240fad90 0x7f8324108900 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 >> 172.21.15.111:0/2256985876 conn(0x7f83240f6530 msgr2=0x7f83240ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 shutdown_connections 2024-02-19T22:12:16.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:16.775+0000 7f8329644700 1 -- 172.21.15.111:0/2256985876 wait complete. 2024-02-19T22:12:17.299 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote config to /etc/ceph/ceph.conf 2024-02-19T22:12:17.299 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:12:17.299 INFO:teuthology.orchestra.run.smithi111.stderr:Creating mgr... 2024-02-19T22:12:17.300 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying port 9283 ... 2024-02-19T22:12:17.301 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl daemon-reload 2024-02-19T22:12:17.395 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov 2024-02-19T22:12:17.424 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl reset-failed ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov 2024-02-19T22:12:17.429 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to reset failed state of unit ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov.service: Unit ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov.service not loaded. 2024-02-19T22:12:17.429 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl enable ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov 2024-02-19T22:12:17.435 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715.target.wants/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov.service → /etc/systemd/system/ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@.service. 2024-02-19T22:12:17.519 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mgr.smithi111.ftjzov 2024-02-19T22:12:18.100 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:16.776117+0000 mon.smithi111 (mon.0) 6 : audit [INF] from='client.? 172.21.15.111:0/2256985876' entity='client.admin' 2024-02-19T22:12:18.103 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:12:18.112 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-02-19T22:12:18.112 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:12:18.119 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:12:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-02-19T22:12:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to enable service . firewalld.service is not available 2024-02-19T22:12:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-02-19T22:12:18.120 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:12:18.128 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-02-19T22:12:18.128 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:12:18.135 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:12:18.136 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-02-19T22:12:18.136 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T22:12:18.136 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr to start... 2024-02-19T22:12:18.137 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr... 2024-02-19T22:12:18.137 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:12:18.705 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.704+0000 7f571feda700 1 Processor -- start 2024-02-19T22:12:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.704+0000 7f571feda700 1 -- start start 2024-02-19T22:12:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.704+0000 7f571feda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.704+0000 7f571feda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180fb6d0 con 0x7f57180fadc0 2024-02-19T22:12:18.707 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571dc76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571dc76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41362/0 (socket says 172.21.15.111:41362) 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571dc76700 1 -- 172.21.15.111:0/407858259 learned_addr learned my addr 172.21.15.111:0/407858259 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571dc76700 1 -- 172.21.15.111:0/407858259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180fb810 con 0x7f57180fadc0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571dc76700 1 --2- 172.21.15.111:0/407858259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f571400b800 tx=0x7f5714006ee0).ready entity=mon.0 client_cookie=47d73c7de82d46b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571cc74700 1 -- 172.21.15.111:0/407858259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5714009740 con 0x7f57180fadc0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571cc74700 1 -- 172.21.15.111:0/407858259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f5714009d30 con 0x7f57180fadc0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.705+0000 7f571cc74700 1 -- 172.21.15.111:0/407858259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5714011e30 con 0x7f57180fadc0 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 -- 172.21.15.111:0/407858259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 msgr2=0x7f57180fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 --2- 172.21.15.111:0/407858259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f571400b800 tx=0x7f5714006ee0).stop 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 -- 172.21.15.111:0/407858259 shutdown_connections 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 --2- 172.21.15.111:0/407858259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f57180fb1e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 -- 172.21.15.111:0/407858259 >> 172.21.15.111:0/407858259 conn(0x7f57180f6520 msgr2=0x7f57180f8940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:18.708 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 -- 172.21.15.111:0/407858259 shutdown_connections 2024-02-19T22:12:18.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.706+0000 7f571feda700 1 -- 172.21.15.111:0/407858259 wait complete. 2024-02-19T22:12:18.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.707+0000 7f571feda700 1 Processor -- start 2024-02-19T22:12:18.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.707+0000 7f571feda700 1 -- start start 2024-02-19T22:12:18.709 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.707+0000 7f571feda700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571feda700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180704b0 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571dc76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571dc76700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:41374/0 (socket says 172.21.15.111:41374) 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571dc76700 1 -- 172.21.15.111:0/3581934830 learned_addr learned my addr 172.21.15.111:0/3581934830 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571dc76700 1 -- 172.21.15.111:0/3581934830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571400b200 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571dc76700 1 --2- 172.21.15.111:0/3581934830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5714002f40 tx=0x7f5714006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f571401a6e0 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f571401acd0 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57180706b0 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f571401bd30 con 0x7f57180fadc0 2024-02-19T22:12:18.710 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.708+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571810df80 con 0x7f57180fadc0 2024-02-19T22:12:18.711 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.709+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f571400f070 con 0x7f57180fadc0 2024-02-19T22:12:18.711 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.709+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5714024c60 con 0x7f57180fadc0 2024-02-19T22:12:18.711 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.710+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5718045ad0 con 0x7f57180fadc0 2024-02-19T22:12:18.712 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.712+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5714016070 con 0x7f57180fadc0 2024-02-19T22:12:18.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.781+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5718027a00 con 0x7f57180fadc0 2024-02-19T22:12:18.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.781+0000 7f57067fc700 1 -- 172.21.15.111:0/3581934830 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f571401ae40 con 0x7f57180fadc0 2024-02-19T22:12:18.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:12:18.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "dbbe4146-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:12:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:12:18.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:12:18.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:11.277955+0000", 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:12:18.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 msgr2=0x7f571810d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 --2- 172.21.15.111:0/3581934830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f5714002f40 tx=0x7f5714006ee0).stop 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 shutdown_connections 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 --2- 172.21.15.111:0/3581934830 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57180fadc0 0x7f571810d850 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 >> 172.21.15.111:0/3581934830 conn(0x7f57180f6520 msgr2=0x7f57180f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 shutdown_connections 2024-02-19T22:12:18.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:18.782+0000 7f571feda700 1 -- 172.21.15.111:0/3581934830 wait complete. 2024-02-19T22:12:19.106 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:18.782124+0000 mon.smithi111 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.111:0/3581934830' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:12:19.359 INFO:teuthology.orchestra.run.smithi111.stderr:mgr not available, waiting (1/15)... 2024-02-19T22:12:21.361 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:12:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.941+0000 7f2c77a16700 1 Processor -- start 2024-02-19T22:12:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.941+0000 7f2c77a16700 1 -- start start 2024-02-19T22:12:21.944 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c77a16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c77a16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700fc8c0 con 0x7f2c700703b0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c757b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c757b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46158/0 (socket says 172.21.15.111:46158) 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c757b2700 1 -- 172.21.15.111:0/3336217329 learned_addr learned my addr 172.21.15.111:0/3336217329 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c757b2700 1 -- 172.21.15.111:0/3336217329 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c701013c0 con 0x7f2c700703b0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c757b2700 1 --2- 172.21.15.111:0/3336217329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c6c00b5f0 tx=0x7f2c6c006ee0).ready entity=mon.0 client_cookie=310594fb06da3723 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c5ffff700 1 -- 172.21.15.111:0/3336217329 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c00d600 con 0x7f2c700703b0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c5ffff700 1 -- 172.21.15.111:0/3336217329 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c6c00dbf0 con 0x7f2c700703b0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.942+0000 7f2c5ffff700 1 -- 172.21.15.111:0/3336217329 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c011cb0 con 0x7f2c700703b0 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.943+0000 7f2c77a16700 1 -- 172.21.15.111:0/3336217329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 msgr2=0x7f2c70100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.943+0000 7f2c77a16700 1 --2- 172.21.15.111:0/3336217329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f2c6c00b5f0 tx=0x7f2c6c006ee0).stop 2024-02-19T22:12:21.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.943+0000 7f2c77a16700 1 -- 172.21.15.111:0/3336217329 shutdown_connections 2024-02-19T22:12:21.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.943+0000 7f2c77a16700 1 --2- 172.21.15.111:0/3336217329 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c70100ed0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:21.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.943+0000 7f2c77a16700 1 -- 172.21.15.111:0/3336217329 >> 172.21.15.111:0/3336217329 conn(0x7f2c700f6530 msgr2=0x7f2c700f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:21.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 -- 172.21.15.111:0/3336217329 shutdown_connections 2024-02-19T22:12:21.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 -- 172.21.15.111:0/3336217329 wait complete. 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 Processor -- start 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 -- start start 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c77a16700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7010d270 con 0x7f2c700703b0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c757b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c757b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46172/0 (socket says 172.21.15.111:46172) 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.944+0000 7f2c757b2700 1 -- 172.21.15.111:0/982727898 learned_addr learned my addr 172.21.15.111:0/982727898 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c757b2700 1 -- 172.21.15.111:0/982727898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c00b200 con 0x7f2c700703b0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c757b2700 1 --2- 172.21.15.111:0/982727898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2c6c006800 tx=0x7f2c6c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c01b570 con 0x7f2c700703b0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f2c6c01bb60 con 0x7f2c700703b0 2024-02-19T22:12:21.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7010d470 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c01a9f0 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7010e080 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f2c6c00f070 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.945+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f2c6c025cb0 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.946+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c70059100 con 0x7f2c700703b0 2024-02-19T22:12:21.949 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:21.948+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f2c6c016070 con 0x7f2c700703b0 2024-02-19T22:12:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.016+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2c701020e0 con 0x7f2c700703b0 2024-02-19T22:12:22.022 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c5dffb700 1 -- 172.21.15.111:0/982727898 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f2c6c01aea0 con 0x7f2c700703b0 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "dbbe4146-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-02-19T22:12:22.023 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:12:22.024 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:12:22.025 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:12:22.026 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.026 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:12:22.026 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": false, 2024-02-19T22:12:22.026 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:12:22.026 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:11.277955+0000", 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:22.028 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 msgr2=0x7f2c7010cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c77a16700 1 --2- 172.21.15.111:0/982727898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f2c6c006800 tx=0x7f2c6c006ee0).stop 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 shutdown_connections 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c77a16700 1 --2- 172.21.15.111:0/982727898 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2c700703b0 0x7f2c7010cd80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.017+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 >> 172.21.15.111:0/982727898 conn(0x7f2c700f6530 msgr2=0x7f2c700f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.018+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 shutdown_connections 2024-02-19T22:12:22.029 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:22.018+0000 7f2c77a16700 1 -- 172.21.15.111:0/982727898 wait complete. 2024-02-19T22:12:22.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.017214+0000 mon.smithi111 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.111:0/982727898' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:12:22.674 INFO:teuthology.orchestra.run.smithi111.stderr:mgr not available, waiting (2/15)... 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:22.939377+0000 mon.smithi111 (mon.0) 9 : cluster [INF] Activating manager daemon smithi111.ftjzov 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:22.941965+0000 mon.smithi111 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi111.ftjzov(active, starting, since 0.00268472s) 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.942998+0000 mon.smithi111 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.943129+0000 mon.smithi111 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.943232+0000 mon.smithi111 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.943383+0000 mon.smithi111 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.943558+0000 mon.smithi111 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mgr metadata", "who": "smithi111.ftjzov", "id": "smithi111.ftjzov"}]: dispatch 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:22.956582+0000 mon.smithi111 (mon.0) 16 : cluster [INF] Manager daemon smithi111.ftjzov is now available 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.964040+0000 mon.smithi111 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.968386+0000 mon.smithi111 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:12:23.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.969477+0000 mon.smithi111 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:12:23.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.974416+0000 mon.smithi111 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:23.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:22.976296+0000 mon.smithi111 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.111:0/3988255800' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:24.676 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-02-19T22:12:25.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:23.945017+0000 mon.smithi111 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi111.ftjzov(active, since 1.00574s) 2024-02-19T22:12:27.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:25.945447+0000 mon.smithi111 (mon.0) 23 : cluster [DBG] mgrmap e4: smithi111.ftjzov(active, since 3s) 2024-02-19T22:12:28.708 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:28.385785+0000 mon.smithi111 (mon.0) 24 : audit [DBG] from='client.? 172.21.15.111:0/2246774371' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff02062b700 1 Processor -- start 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff02062b700 1 -- start start 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff02062b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff02062b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180fb6a0 con 0x7ff0180fad90 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff01e3c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff01e3c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46228/0 (socket says 172.21.15.111:46228) 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff01e3c7700 1 -- 172.21.15.111:0/1363041653 learned_addr learned my addr 172.21.15.111:0/1363041653 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.173+0000 7ff01e3c7700 1 -- 172.21.15.111:0/1363041653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180fb7e0 con 0x7ff0180fad90 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff01e3c7700 1 --2- 172.21.15.111:0/1363041653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff00800b5f0 tx=0x7ff008006ee0).ready entity=mon.0 client_cookie=9a2f63f4d1dbad85 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:28.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff01d3c5700 1 -- 172.21.15.111:0/1363041653 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff00800d600 con 0x7ff0180fad90 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff01d3c5700 1 -- 172.21.15.111:0/1363041653 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff00800dbf0 con 0x7ff0180fad90 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff01d3c5700 1 -- 172.21.15.111:0/1363041653 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff008011cb0 con 0x7ff0180fad90 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 -- 172.21.15.111:0/1363041653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 msgr2=0x7ff0180fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 --2- 172.21.15.111:0/1363041653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff00800b5f0 tx=0x7ff008006ee0).stop 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 -- 172.21.15.111:0/1363041653 shutdown_connections 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 --2- 172.21.15.111:0/1363041653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff0180fb1b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 -- 172.21.15.111:0/1363041653 >> 172.21.15.111:0/1363041653 conn(0x7ff0180f6530 msgr2=0x7ff0180f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 -- 172.21.15.111:0/1363041653 shutdown_connections 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.174+0000 7ff02062b700 1 -- 172.21.15.111:0/1363041653 wait complete. 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff02062b700 1 Processor -- start 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff02062b700 1 -- start start 2024-02-19T22:12:28.911 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff02062b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:28.912 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff02062b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180ff4c0 con 0x7ff0180fad90 2024-02-19T22:12:28.912 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff01e3c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff01e3c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:46234/0 (socket says 172.21.15.111:46234) 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.175+0000 7ff01e3c7700 1 -- 172.21.15.111:0/2246774371 learned_addr learned my addr 172.21.15.111:0/2246774371 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff01e3c7700 1 -- 172.21.15.111:0/2246774371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00800b200 con 0x7ff0180fad90 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff01e3c7700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff008002f40 tx=0x7ff008006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff00801b410 con 0x7ff0180fad90 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff00801ba00 con 0x7ff0180fad90 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff00801a900 con 0x7ff0180fad90 2024-02-19T22:12:28.921 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0180ff660 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.176+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0180ffaf0 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7ff00800f070 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff006ffd700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 0x7ff0000322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff008041390 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff01dbc6700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 0x7ff0000322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff018059100 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.177+0000 7ff01dbc6700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 0x7ff0000322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.182+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff008016030 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.385+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff018045ad0 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.385+0000 7ff006ffd700 1 -- 172.21.15.111:0/2246774371 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7ff008033090 con 0x7ff0180fad90 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 msgr2=0x7ff0000322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 0x7ff0000322d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff0140096a0 tx=0x7ff014006b40).stop 2024-02-19T22:12:28.922 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 msgr2=0x7ff018101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7ff008002f40 tx=0x7ff008006ee0).stop 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 shutdown_connections 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff00002fe10 0x7ff0000322d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 --2- 172.21.15.111:0/2246774371 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0180fad90 0x7ff018101040 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 >> 172.21.15.111:0/2246774371 conn(0x7ff0180f6530 msgr2=0x7ff018070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 shutdown_connections 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.387+0000 7ff02062b700 1 -- 172.21.15.111:0/2246774371 wait complete. 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "dbbe4146-cf73-11ee-95bb-87774f69a715", 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-02-19T22:12:28.923 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-02-19T22:12:28.924 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-02-19T22:12:28.925 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-02-19T22:12:28.926 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-02-19T22:12:11.277955+0000", 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-02-19T22:12:28.927 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:12:33.470 INFO:teuthology.orchestra.run.smithi111.stderr:mgr is available 2024-02-19T22:12:33.471 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling cephadm module... 2024-02-19T22:12:33.471 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-02-19T22:12:36.218 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.214+0000 7fa57a237700 1 Processor -- start 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.214+0000 7fa57a237700 1 -- start start 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.214+0000 7fa57a237700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.214+0000 7fa57a237700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5740fed80 con 0x7fa574070430 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa579235700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa579235700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49148/0 (socket says 172.21.15.111:49148) 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa579235700 1 -- 172.21.15.111:0/1030017601 learned_addr learned my addr 172.21.15.111:0/1030017601 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa579235700 1 -- 172.21.15.111:0/1030017601 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5740feec0 con 0x7fa574070430 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa579235700 1 --2- 172.21.15.111:0/1030017601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa56400b5f0 tx=0x7fa564006ee0).ready entity=mon.0 client_cookie=2783fb2475aaf1e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa563fff700 1 -- 172.21.15.111:0/1030017601 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa56400d600 con 0x7fa574070430 2024-02-19T22:12:36.219 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa563fff700 1 -- 172.21.15.111:0/1030017601 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa56400dbf0 con 0x7fa574070430 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.215+0000 7fa563fff700 1 -- 172.21.15.111:0/1030017601 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa564011cb0 con 0x7fa574070430 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 -- 172.21.15.111:0/1030017601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 msgr2=0x7fa5740fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 --2- 172.21.15.111:0/1030017601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fa56400b5f0 tx=0x7fa564006ee0).stop 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 -- 172.21.15.111:0/1030017601 shutdown_connections 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 --2- 172.21.15.111:0/1030017601 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa5740fe890 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 -- 172.21.15.111:0/1030017601 >> 172.21.15.111:0/1030017601 conn(0x7fa5740f6390 msgr2=0x7fa5740f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 -- 172.21.15.111:0/1030017601 shutdown_connections 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.216+0000 7fa57a237700 1 -- 172.21.15.111:0/1030017601 wait complete. 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa57a237700 1 Processor -- start 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa57a237700 1 -- start start 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa57a237700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa57a237700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57410ae80 con 0x7fa574070430 2024-02-19T22:12:36.220 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa579235700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa579235700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49158/0 (socket says 172.21.15.111:49158) 2024-02-19T22:12:36.221 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa579235700 1 -- 172.21.15.111:0/3681016663 learned_addr learned my addr 172.21.15.111:0/3681016663 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa579235700 1 -- 172.21.15.111:0/3681016663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56400b200 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.217+0000 7fa579235700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa564006800 tx=0x7fa564006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.218+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa56401b570 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.218+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fa56401bb60 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.218+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa56401a9f0 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.218+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57410b080 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.218+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57410b510 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fa56400f070 con 0x7fa574070430 2024-02-19T22:12:36.222 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa561ffb700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 0x7fa55c032280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa564041400 con 0x7fa574070430 2024-02-19T22:12:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa578a34700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 0x7fa55c032280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa574045ad0 con 0x7fa574070430 2024-02-19T22:12:36.223 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.219+0000 7fa578a34700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 0x7fa55c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa5700096a0 tx=0x7fa570006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:36.224 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.223+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa564016070 con 0x7fa574070430 2024-02-19T22:12:36.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:36.407+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fa574104960 con 0x7fa574070430 2024-02-19T22:12:37.150 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.149+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa56401bcd0 con 0x7fa574070430 2024-02-19T22:12:37.151 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.150+0000 7fa561ffb700 1 -- 172.21.15.111:0/3681016663 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fa5640239f0 con 0x7fa574070430 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 msgr2=0x7fa55c032280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 0x7fa55c032280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa5700096a0 tx=0x7fa570006b40).stop 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 msgr2=0x7fa57410a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fa564006800 tx=0x7fa564006ee0).stop 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 shutdown_connections 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7fa55c02fdc0 0x7fa55c032280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 --2- 172.21.15.111:0/3681016663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa574070430 0x7fa57410a990 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 >> 172.21.15.111:0/3681016663 conn(0x7fa5740f6390 msgr2=0x7fa5740f7d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:37.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 shutdown_connections 2024-02-19T22:12:37.155 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:37.152+0000 7fa57a237700 1 -- 172.21.15.111:0/3681016663 wait complete. 2024-02-19T22:12:37.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:37 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:36.407681+0000 mon.smithi111 (mon.0) 25 : audit [INF] from='client.? 172.21.15.111:0/3681016663' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T22:12:37.832 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:12:38.352 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.348+0000 7f9abc5aa700 1 Processor -- start 2024-02-19T22:12:38.352 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.348+0000 7f9abc5aa700 1 -- start start 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9abc5aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9abc5aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab40fc8c0 con 0x7f9ab40703b0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9aba346700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9aba346700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49182/0 (socket says 172.21.15.111:49182) 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9aba346700 1 -- 172.21.15.111:0/1052410136 learned_addr learned my addr 172.21.15.111:0/1052410136 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9aba346700 1 -- 172.21.15.111:0/1052410136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab41013c0 con 0x7f9ab40703b0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.349+0000 7f9aba346700 1 --2- 172.21.15.111:0/1052410136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f9ab000b5f0 tx=0x7f9ab0006ee0).ready entity=mon.0 client_cookie=ad5decf22d6912f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9ab9344700 1 -- 172.21.15.111:0/1052410136 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab000d600 con 0x7f9ab40703b0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9ab9344700 1 -- 172.21.15.111:0/1052410136 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9ab000dbf0 con 0x7f9ab40703b0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9ab9344700 1 -- 172.21.15.111:0/1052410136 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab0011cb0 con 0x7f9ab40703b0 2024-02-19T22:12:38.353 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 -- 172.21.15.111:0/1052410136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 msgr2=0x7f9ab4100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/1052410136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f9ab000b5f0 tx=0x7f9ab0006ee0).stop 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 -- 172.21.15.111:0/1052410136 shutdown_connections 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/1052410136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab4100ed0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 -- 172.21.15.111:0/1052410136 >> 172.21.15.111:0/1052410136 conn(0x7f9ab40f6530 msgr2=0x7f9ab40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 -- 172.21.15.111:0/1052410136 shutdown_connections 2024-02-19T22:12:38.354 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.350+0000 7f9abc5aa700 1 -- 172.21.15.111:0/1052410136 wait complete. 2024-02-19T22:12:38.356 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9abc5aa700 1 Processor -- start 2024-02-19T22:12:38.356 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9abc5aa700 1 -- start start 2024-02-19T22:12:38.356 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9abc5aa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9abc5aa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab410d2c0 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9aba346700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9aba346700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49188/0 (socket says 172.21.15.111:49188) 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.351+0000 7f9aba346700 1 -- 172.21.15.111:0/2347523557 learned_addr learned my addr 172.21.15.111:0/2347523557 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9aba346700 1 -- 172.21.15.111:0/2347523557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab000b200 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9aba346700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9ab000df90 tx=0x7f9ab0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab001b410 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f9ab001ba00 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab410d4c0 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ab001aa30 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.352+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab410e100 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9ab000f070 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9aa2ffd700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 0x7f9a9c032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9ab00410e0 con 0x7f9ab40703b0 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9ab9b45700 1 -- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 msgr2=0x7f9a9c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:38.357 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9ab9b45700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 0x7f9a9c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:12:38.358 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.353+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab410d950 con 0x7f9ab40703b0 2024-02-19T22:12:38.359 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.357+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ab0016070 con 0x7f9ab40703b0 2024-02-19T22:12:38.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:37.150575+0000 mon.smithi111 (mon.0) 26 : audit [INF] from='client.? 172.21.15.111:0/3681016663' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T22:12:38.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:37.150624+0000 mon.smithi111 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi111.ftjzov(active, since 14s) 2024-02-19T22:12:38.556 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.554+0000 7f9ab9b45700 1 -- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 msgr2=0x7f9a9c032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:38.556 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.554+0000 7f9ab9b45700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 0x7f9a9c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:12:38.589 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.587+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9ab41020e0 con 0x7f9ab40703b0 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.587+0000 7f9aa2ffd700 1 -- 172.21.15.111:0/2347523557 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9ab0022d50 con 0x7f9ab40703b0 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 5, 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "active_name": "smithi111.ftjzov", 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:12:38.590 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 msgr2=0x7f9a9c032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 0x7f9a9c032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 msgr2=0x7f9ab410cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f9ab000df90 tx=0x7f9ab0006ee0).stop 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 shutdown_connections 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7f9a9c02fe70 0x7f9a9c032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:12:38.592 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 --2- 172.21.15.111:0/2347523557 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9ab40703b0 0x7f9ab410cdd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:38.593 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 >> 172.21.15.111:0/2347523557 conn(0x7f9ab40f6530 msgr2=0x7f9ab40f7fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:38.593 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 shutdown_connections 2024-02-19T22:12:38.593 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:38.589+0000 7f9abc5aa700 1 -- 172.21.15.111:0/2347523557 wait complete. 2024-02-19T22:12:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for the mgr to restart... 2024-02-19T22:12:39.205 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr epoch 5... 2024-02-19T22:12:39.206 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:12:39.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:39 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:38.588751+0000 mon.smithi111 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.111:0/2347523557' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:12:39.749 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.747+0000 7ff22ba3d700 1 Processor -- start 2024-02-19T22:12:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.747+0000 7ff22ba3d700 1 -- start start 2024-02-19T22:12:39.750 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff22ba3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:39.752 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff22ba3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2240fb6a0 con 0x7ff2240fad90 2024-02-19T22:12:39.752 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff2297d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff2297d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49194/0 (socket says 172.21.15.111:49194) 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff2297d9700 1 -- 172.21.15.111:0/2808043724 learned_addr learned my addr 172.21.15.111:0/2808043724 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff2297d9700 1 -- 172.21.15.111:0/2808043724 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2240fb7e0 con 0x7ff2240fad90 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff2297d9700 1 --2- 172.21.15.111:0/2808043724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff21400b5f0 tx=0x7ff214006ee0).ready entity=mon.0 client_cookie=9f3ba1c80b08cc56 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff213fff700 1 -- 172.21.15.111:0/2808043724 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff21400d600 con 0x7ff2240fad90 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff213fff700 1 -- 172.21.15.111:0/2808043724 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff21400dbf0 con 0x7ff2240fad90 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.748+0000 7ff213fff700 1 -- 172.21.15.111:0/2808043724 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff214011cb0 con 0x7ff2240fad90 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 -- 172.21.15.111:0/2808043724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 msgr2=0x7ff2240fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/2808043724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7ff21400b5f0 tx=0x7ff214006ee0).stop 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 -- 172.21.15.111:0/2808043724 shutdown_connections 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/2808043724 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff2240fb1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 -- 172.21.15.111:0/2808043724 >> 172.21.15.111:0/2808043724 conn(0x7ff2240f6530 msgr2=0x7ff2240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 -- 172.21.15.111:0/2808043724 shutdown_connections 2024-02-19T22:12:39.753 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.749+0000 7ff22ba3d700 1 -- 172.21.15.111:0/2808043724 wait complete. 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff22ba3d700 1 Processor -- start 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff22ba3d700 1 -- start start 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff22ba3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff22ba3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22410b020 con 0x7ff2240fad90 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff2297d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff2297d9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:49210/0 (socket says 172.21.15.111:49210) 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff2297d9700 1 -- 172.21.15.111:0/3171148583 learned_addr learned my addr 172.21.15.111:0/3171148583 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff2297d9700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21400b200 con 0x7ff2240fad90 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.750+0000 7ff2297d9700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff214011ea0 tx=0x7ff214006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.751+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff21401b570 con 0x7ff2240fad90 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.751+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22410b220 con 0x7ff2240fad90 2024-02-19T22:12:39.754 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.751+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff21401bb60 con 0x7ff2240fad90 2024-02-19T22:12:39.755 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.751+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff21401a8d0 con 0x7ff2240fad90 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.751+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22410b6b0 con 0x7ff2240fad90 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff21400f070 con 0x7ff2240fad90 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff211ffb700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff20c032990 con 0x7ff20c02fe10 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7ff2140414c0 con 0x7ff2240fad90 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff228fd8700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 msgr2=0x7ff20c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:39.756 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.752+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:12:39.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.953+0000 7ff228fd8700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 msgr2=0x7ff20c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:39.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:39.953+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:12:40.356 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:40.354+0000 7ff228fd8700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 msgr2=0x7ff20c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:40.356 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:40.354+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:12:41.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:41.155+0000 7ff228fd8700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 msgr2=0x7ff20c0322d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/2511749600 2024-02-19T22:12:41.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:41.155+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:12:42.150 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:42.148+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7ff21401ab90 con 0x7ff2240fad90 2024-02-19T22:12:42.150 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:42.148+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 msgr2=0x7ff20c0322d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:12:42.150 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:42.148+0000 7ff211ffb700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:12:42.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:42.080371+0000 mon.smithi111 (mon.0) 29 : cluster [INF] Active manager daemon smithi111.ftjzov restarted 2024-02-19T22:12:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:42.080739+0000 mon.smithi111 (mon.0) 30 : cluster [INF] Activating manager daemon smithi111.ftjzov 2024-02-19T22:12:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:42.082439+0000 mon.smithi111 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-02-19T22:12:43.155 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.152+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7ff214024ca0 con 0x7ff2240fad90 2024-02-19T22:12:43.155 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.152+0000 7ff211ffb700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 0x7ff20c035cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:43.155 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.152+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff20c032990 con 0x7ff20c0338e0 2024-02-19T22:12:43.156 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.153+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 0x7ff20c035cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:43.156 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.154+0000 7ff228fd8700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 0x7ff20c035cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff2200036d0 tx=0x7ff220006ca0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:43.156 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.154+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7ff20c032990 con 0x7ff20c0338e0 2024-02-19T22:12:43.162 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff224059100 con 0x7ff20c0338e0 2024-02-19T22:12:43.162 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff211ffb700 1 -- 172.21.15.111:0/3171148583 <== mgr.14114 v2:172.21.15.111:6800/3561110342 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff224059100 con 0x7ff20c0338e0 2024-02-19T22:12:43.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 msgr2=0x7ff20c035cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 0x7ff20c035cd0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7ff2200036d0 tx=0x7ff220006ca0).stop 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 msgr2=0x7ff22410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:43.164 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7ff214011ea0 tx=0x7ff214006ee0).stop 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 shutdown_connections 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff20c0338e0 0x7ff20c035cd0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2240fad90 0x7ff22410ab30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 --2- 172.21.15.111:0/3171148583 >> [v2:172.21.15.111:6800/2511749600,v1:172.21.15.111:6801/2511749600] conn(0x7ff20c02fe10 0x7ff20c0322d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 >> 172.21.15.111:0/3171148583 conn(0x7ff2240f6530 msgr2=0x7ff2240700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.161+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 shutdown_connections 2024-02-19T22:12:43.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:43.162+0000 7ff22ba3d700 1 -- 172.21.15.111:0/3171148583 wait complete. 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:42.150777+0000 mon.smithi111 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi111.ftjzov(active, starting, since 0.0701093s) 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.151738+0000 mon.smithi111 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.152463+0000 mon.smithi111 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mgr metadata", "who": "smithi111.ftjzov", "id": "smithi111.ftjzov"}]: dispatch 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.153095+0000 mon.smithi111 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.153244+0000 mon.smithi111 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.153389+0000 mon.smithi111 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:12:43.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:42.166630+0000 mon.smithi111 (mon.0) 38 : cluster [INF] Manager daemon smithi111.ftjzov is now available 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.179480+0000 mon.smithi111 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.182308+0000 mon.smithi111 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.184786+0000 mon.smithi111 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.272371+0000 mon.smithi111 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.273033+0000 mon.smithi111 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.274470+0000 mon.smithi111 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.275932+0000 mon.smithi111 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.286189+0000 mon.smithi111 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:12:43.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:42.287274+0000 mon.smithi111 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:12:43.721 INFO:teuthology.orchestra.run.smithi111.stderr:mgr epoch 5 is available 2024-02-19T22:12:43.721 INFO:teuthology.orchestra.run.smithi111.stderr:Setting orchestrator backend to cephadm... 2024-02-19T22:12:43.721 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-02-19T22:12:44.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.283+0000 7fcbe268e700 1 Processor -- start 2024-02-19T22:12:44.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe268e700 1 -- start start 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe268e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe268e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc0fc6c0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35082/0 (socket says 172.21.15.111:35082) 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe168c700 1 -- 172.21.15.111:0/886617775 learned_addr learned my addr 172.21.15.111:0/886617775 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe168c700 1 -- 172.21.15.111:0/886617775 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc1011c0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.284+0000 7fcbe168c700 1 --2- 172.21.15.111:0/886617775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fcbcc00b5f0 tx=0x7fcbcc006ee0).ready entity=mon.0 client_cookie=f58436eaa055ad42 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbcbfff700 1 -- 172.21.15.111:0/886617775 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbcc00d600 con 0x7fcbdc0703f0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbcbfff700 1 -- 172.21.15.111:0/886617775 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcbcc00dbf0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbcbfff700 1 -- 172.21.15.111:0/886617775 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbcc011ce0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 -- 172.21.15.111:0/886617775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 msgr2=0x7fcbdc100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 --2- 172.21.15.111:0/886617775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fcbcc00b5f0 tx=0x7fcbcc006ee0).stop 2024-02-19T22:12:44.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 -- 172.21.15.111:0/886617775 shutdown_connections 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 --2- 172.21.15.111:0/886617775 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc100cd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 -- 172.21.15.111:0/886617775 >> 172.21.15.111:0/886617775 conn(0x7fcbdc0f6390 msgr2=0x7fcbdc0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 -- 172.21.15.111:0/886617775 shutdown_connections 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.285+0000 7fcbe268e700 1 -- 172.21.15.111:0/886617775 wait complete. 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe268e700 1 Processor -- start 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe268e700 1 -- start start 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe268e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe268e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc10d070 con 0x7fcbdc0703f0 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe168c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35092/0 (socket says 172.21.15.111:35092) 2024-02-19T22:12:44.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe168c700 1 -- 172.21.15.111:0/1545864583 learned_addr learned my addr 172.21.15.111:0/1545864583 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:44.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.286+0000 7fcbe168c700 1 -- 172.21.15.111:0/1545864583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbcc00b200 con 0x7fcbdc0703f0 2024-02-19T22:12:44.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbe168c700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcbcc011ed0 tx=0x7fcbcc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbcc01b570 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcbcc01bb60 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc10d270 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbcc01aad0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.287+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc10d700 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.288+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fcbcc00f070 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.288+0000 7fcbc9ffb700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 0x7fcbc4032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.288+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc045ad0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.288+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcbcc041420 con 0x7fcbdc0703f0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.288+0000 7fcbe0e8b700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 0x7fcbc4032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:44.292 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.289+0000 7fcbe0e8b700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 0x7fcbc4032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcbd80096a0 tx=0x7fcbd8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:44.294 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.292+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbcc0228b0 con 0x7fcbdc0703f0 2024-02-19T22:12:44.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:43.155737+0000 mgr.smithi111.ftjzov (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:12:44.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:43.156226+0000 mon.smithi111 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi111.ftjzov(active, since 1.07555s) 2024-02-19T22:12:44.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:43.162539+0000 mgr.smithi111.ftjzov (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:12:44.473 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.471+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbdc10df60 con 0x7fcbc402fe70 2024-02-19T22:12:44.478 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.476+0000 7fcbc9ffb700 1 -- 172.21.15.111:0/1545864583 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcbdc10df60 con 0x7fcbc402fe70 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 msgr2=0x7fcbc4032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 0x7fcbc4032330 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcbd80096a0 tx=0x7fcbd8006b40).stop 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 msgr2=0x7fcbdc10cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcbcc011ed0 tx=0x7fcbcc006ee0).stop 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 shutdown_connections 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcbc402fe70 0x7fcbc4032330 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 --2- 172.21.15.111:0/1545864583 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcbdc0703f0 0x7fcbdc10cb80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 >> 172.21.15.111:0/1545864583 conn(0x7fcbdc0f6390 msgr2=0x7fcbdc0f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 shutdown_connections 2024-02-19T22:12:44.481 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:44.478+0000 7fcbe268e700 1 -- 172.21.15.111:0/1545864583 wait complete. 2024-02-19T22:12:45.102 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-02-19T22:12:45.370 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:44.157749+0000 mon.smithi111 (mon.0) 49 : cluster [DBG] mgrmap e8: smithi111.ftjzov(active, since 2s) 2024-02-19T22:12:45.371 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:44.477619+0000 mon.smithi111 (mon.0) 50 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:45.371 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:44.481132+0000 mon.smithi111 (mon.0) 51 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.628+0000 7f6c49914700 1 Processor -- start 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.628+0000 7f6c49914700 1 -- start start 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c49914700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c49914700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c440fc7b0 con 0x7f6c440fbea0 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35098/0 (socket says 172.21.15.111:35098) 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c42ffd700 1 -- 172.21.15.111:0/2492421932 learned_addr learned my addr 172.21.15.111:0/2492421932 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c42ffd700 1 -- 172.21.15.111:0/2492421932 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c440fc8f0 con 0x7f6c440fbea0 2024-02-19T22:12:45.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c42ffd700 1 --2- 172.21.15.111:0/2492421932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6c3400b5f0 tx=0x7f6c34006ee0).ready entity=mon.0 client_cookie=521502687a5258b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c41ffb700 1 -- 172.21.15.111:0/2492421932 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3400d600 con 0x7f6c440fbea0 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c41ffb700 1 -- 172.21.15.111:0/2492421932 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6c3400dbf0 con 0x7f6c440fbea0 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.629+0000 7f6c41ffb700 1 -- 172.21.15.111:0/2492421932 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c34011cb0 con 0x7f6c440fbea0 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 -- 172.21.15.111:0/2492421932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 msgr2=0x7f6c440fc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 --2- 172.21.15.111:0/2492421932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6c3400b5f0 tx=0x7f6c34006ee0).stop 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 -- 172.21.15.111:0/2492421932 shutdown_connections 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 --2- 172.21.15.111:0/2492421932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440fc2c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 -- 172.21.15.111:0/2492421932 >> 172.21.15.111:0/2492421932 conn(0x7f6c440f76a0 msgr2=0x7f6c440f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 -- 172.21.15.111:0/2492421932 shutdown_connections 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.630+0000 7f6c49914700 1 -- 172.21.15.111:0/2492421932 wait complete. 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c49914700 1 Processor -- start 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c49914700 1 -- start start 2024-02-19T22:12:45.635 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c49914700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c49914700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44070a60 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c42ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35100/0 (socket says 172.21.15.111:35100) 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c42ffd700 1 -- 172.21.15.111:0/1948977425 learned_addr learned my addr 172.21.15.111:0/1948977425 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c42ffd700 1 -- 172.21.15.111:0/1948977425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3400b200 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c42ffd700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6c3400bf40 tx=0x7f6c34006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.631+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3401b570 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f6c3401bb60 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c3401aa20 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c44070c00 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c44071090 con 0x7f6c440fbea0 2024-02-19T22:12:45.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6c3400f070 con 0x7f6c440fbea0 2024-02-19T22:12:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.632+0000 7f6c3b7fe700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 0x7f6c240322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.633+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6c340414a0 con 0x7f6c440fbea0 2024-02-19T22:12:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.633+0000 7f6c427fc700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 0x7f6c240322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:45.637 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.633+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c44059100 con 0x7f6c440fbea0 2024-02-19T22:12:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.633+0000 7f6c427fc700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 0x7f6c240322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6c2c0096a0 tx=0x7f6c2c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:45.638 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.637+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c34016070 con 0x7f6c440fbea0 2024-02-19T22:12:45.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.788+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c44104b80 con 0x7f6c2402fe10 2024-02-19T22:12:45.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.789+0000 7f6c3b7fe700 1 -- 172.21.15.111:0/1948977425 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6c44104b80 con 0x7f6c2402fe10 2024-02-19T22:12:45.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: value unchanged 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.790+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 msgr2=0x7f6c240322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.790+0000 7f6c49914700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 0x7f6c240322d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6c2c0096a0 tx=0x7f6c2c006b40).stop 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.790+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 msgr2=0x7f6c440725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.790+0000 7f6c49914700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6c3400bf40 tx=0x7f6c34006ee0).stop 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 shutdown_connections 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f6c2402fe10 0x7f6c240322d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 --2- 172.21.15.111:0/1948977425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6c440fbea0 0x7f6c440725e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 >> 172.21.15.111:0/1948977425 conn(0x7f6c440f76a0 msgr2=0x7f6c440f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 shutdown_connections 2024-02-19T22:12:45.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:45.791+0000 7f6c49914700 1 -- 172.21.15.111:0/1948977425 wait complete. 2024-02-19T22:12:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:44.473847+0000 mgr.smithi111.ftjzov (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:46.526 INFO:teuthology.orchestra.run.smithi111.stderr:Generating ssh key... 2024-02-19T22:12:46.526 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-02-19T22:12:47.068 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.065+0000 7f50ea8eb700 1 Processor -- start 2024-02-19T22:12:47.068 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.065+0000 7f50ea8eb700 1 -- start start 2024-02-19T22:12:47.068 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.065+0000 7f50ea8eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:47.068 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.065+0000 7f50ea8eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40fefa0 con 0x7f50e40703d0 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35106/0 (socket says 172.21.15.111:35106) 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e3fff700 1 -- 172.21.15.111:0/3126520003 learned_addr learned my addr 172.21.15.111:0/3126520003 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e3fff700 1 -- 172.21.15.111:0/3126520003 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40ff0e0 con 0x7f50e40703d0 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e3fff700 1 --2- 172.21.15.111:0/3126520003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f50d400b5f0 tx=0x7f50d4006ee0).ready entity=mon.0 client_cookie=e306b77b0fd8b590 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:47.069 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e2ffd700 1 -- 172.21.15.111:0/3126520003 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d400d600 con 0x7f50e40703d0 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e2ffd700 1 -- 172.21.15.111:0/3126520003 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f50d400dbf0 con 0x7f50e40703d0 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.066+0000 7f50e2ffd700 1 -- 172.21.15.111:0/3126520003 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4011cb0 con 0x7f50e40703d0 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- 172.21.15.111:0/3126520003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 msgr2=0x7f50e40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/3126520003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f50d400b5f0 tx=0x7f50d4006ee0).stop 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- 172.21.15.111:0/3126520003 shutdown_connections 2024-02-19T22:12:47.070 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/3126520003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e40feab0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- 172.21.15.111:0/3126520003 >> 172.21.15.111:0/3126520003 conn(0x7f50e40f6530 msgr2=0x7f50e40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- 172.21.15.111:0/3126520003 shutdown_connections 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- 172.21.15.111:0/3126520003 wait complete. 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 Processor -- start 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.067+0000 7f50ea8eb700 1 -- start start 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50ea8eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50ea8eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e410b060 con 0x7f50e40703d0 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35112/0 (socket says 172.21.15.111:35112) 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e3fff700 1 -- 172.21.15.111:0/1990274963 learned_addr learned my addr 172.21.15.111:0/1990274963 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:47.071 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e3fff700 1 -- 172.21.15.111:0/1990274963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d400b200 con 0x7f50e40703d0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e3fff700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f50d4006a20 tx=0x7f50d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d401b570 con 0x7f50e40703d0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f50d401bb60 con 0x7f50e40703d0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e410b260 con 0x7f50e40703d0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d401aa20 con 0x7f50e40703d0 2024-02-19T22:12:47.072 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.068+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e410b6f0 con 0x7f50e40703d0 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.069+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f50d400f070 con 0x7f50e40703d0 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.069+0000 7f50e0ff9700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 0x7f50d0032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.069+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f50d4041280 con 0x7f50e40703d0 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.069+0000 7f50e37fe700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 0x7f50d0032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.069+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e4045ad0 con 0x7f50e40703d0 2024-02-19T22:12:47.073 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.070+0000 7f50e37fe700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 0x7f50d0032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f50d80096a0 tx=0x7f50d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:47.075 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.074+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50d4022d50 con 0x7f50e40703d0 2024-02-19T22:12:47.229 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.227+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f50e4104bc0 con 0x7f50d002fe70 2024-02-19T22:12:47.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:47 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:45.790474+0000 mgr.smithi111.ftjzov (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:47.518 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.516+0000 7f50e0ff9700 1 -- 172.21.15.111:0/1990274963 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f50e4104bc0 con 0x7f50d002fe70 2024-02-19T22:12:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 msgr2=0x7f50d0032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:47.520 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 0x7f50d0032330 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f50d80096a0 tx=0x7f50d8006b40).stop 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 msgr2=0x7f50e410ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f50d4006a20 tx=0x7f50d4006ee0).stop 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 shutdown_connections 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f50d002fe70 0x7f50d0032330 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 --2- 172.21.15.111:0/1990274963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50e40703d0 0x7f50e410ab70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 >> 172.21.15.111:0/1990274963 conn(0x7f50e40f6530 msgr2=0x7f50e40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 shutdown_connections 2024-02-19T22:12:47.521 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:47.518+0000 7f50ea8eb700 1 -- 172.21.15.111:0/1990274963 wait complete. 2024-02-19T22:12:48.091 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f663bd47700 1 Processor -- start 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f663bd47700 1 -- start start 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f663bd47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f663bd47700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66340fef80 con 0x7f66340703f0 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f6639ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:48.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f6639ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35118/0 (socket says 172.21.15.111:35118) 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f6639ae3700 1 -- 172.21.15.111:0/2046034966 learned_addr learned my addr 172.21.15.111:0/2046034966 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.624+0000 7f6639ae3700 1 -- 172.21.15.111:0/2046034966 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66340ff0c0 con 0x7f66340703f0 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f6639ae3700 1 --2- 172.21.15.111:0/2046034966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f663000b5f0 tx=0x7f6630006ee0).ready entity=mon.0 client_cookie=520b0907dcd10f11 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f6638ae1700 1 -- 172.21.15.111:0/2046034966 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663000d600 con 0x7f66340703f0 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f6638ae1700 1 -- 172.21.15.111:0/2046034966 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f663000dbf0 con 0x7f66340703f0 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f6638ae1700 1 -- 172.21.15.111:0/2046034966 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6630011ce0 con 0x7f66340703f0 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 -- 172.21.15.111:0/2046034966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 msgr2=0x7f66340fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 --2- 172.21.15.111:0/2046034966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f663000b5f0 tx=0x7f6630006ee0).stop 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 -- 172.21.15.111:0/2046034966 shutdown_connections 2024-02-19T22:12:48.631 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 --2- 172.21.15.111:0/2046034966 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f66340fea90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 -- 172.21.15.111:0/2046034966 >> 172.21.15.111:0/2046034966 conn(0x7f66340f6530 msgr2=0x7f66340f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.625+0000 7f663bd47700 1 -- 172.21.15.111:0/2046034966 shutdown_connections 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f663bd47700 1 -- 172.21.15.111:0/2046034966 wait complete. 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f663bd47700 1 Processor -- start 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f663bd47700 1 -- start start 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f663bd47700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f663bd47700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663410b080 con 0x7f66340703f0 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f6639ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f6639ae3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35134/0 (socket says 172.21.15.111:35134) 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.626+0000 7f6639ae3700 1 -- 172.21.15.111:0/2947752071 learned_addr learned my addr 172.21.15.111:0/2947752071 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f6639ae3700 1 -- 172.21.15.111:0/2947752071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663000b200 con 0x7f66340703f0 2024-02-19T22:12:48.632 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f6639ae3700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f66300115c0 tx=0x7f6630006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663001b410 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f663001ba00 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f663001aaa0 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663410b280 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.627+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663410b710 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f663000f070 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f66227fc700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 0x7f661c0322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6630041000 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f66392e2700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 0x7f661c0322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6634045ad0 con 0x7f66340703f0 2024-02-19T22:12:48.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.628+0000 7f66392e2700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 0x7f661c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f66240096a0 tx=0x7f6624006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:48.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.632+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6630016070 con 0x7f66340703f0 2024-02-19T22:12:48.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.782+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6634104c00 con 0x7f661c02fe10 2024-02-19T22:12:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.783+0000 7f66227fc700 1 -- 172.21.15.111:0/2947752071 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f6634104c00 con 0x7f661c02fe10 2024-02-19T22:12:48.785 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQJ8pRcYDMY37GE3N9RXLEW/1HWKkelnfNzrVYuPeLt1Y3SMcUgx0ZTrK+MYsr7Jfik1mCB1wLj8oqM38HsZ1Djg0CHDqDwOxDG+MRvLEvlSV6hYVuGjeWZdkuJBNh3LNiSI4/bpPTfZ0k6odqQZ5xjQd47+vqTgMqm7a/4MYp5n0xG5EWhi9/dvfCl1dmb4F26anEeF1Zv7aYCPFFov/irslg/8NLJR4TPk+QvmsSHX077q5fxZ31KZYRbpctveoYSs0Ay3LZHVLOnPt0pgz1giHXyo6/kqu1OhGqwXNa2bRgm/bh37BFVoTe+DPF110eD2wFssfC/av5j6p95xNmy/JBTA9VGKzMezU03KgAF3C+lmPXAbGsbHEn9mva9uatsu+I0yzlK3Ei4oUlb0lNFQuHdFt1y5jxJRzKpNfFUadmzR4lI8QKzPa6rF+DNrcX8Q8TSZ8caXdZ/bNFN4/8oFaKMsxxc2IqA6xOX4kvnrZByrkEX61VMTg1pvyi/IM= ceph-dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:12:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 msgr2=0x7f661c0322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 0x7f661c0322d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f66240096a0 tx=0x7f6624006b40).stop 2024-02-19T22:12:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 msgr2=0x7f663410ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:48.787 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f66300115c0 tx=0x7f6630006ee0).stop 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 shutdown_connections 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f661c02fe10 0x7f661c0322d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 --2- 172.21.15.111:0/2947752071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f66340703f0 0x7f663410ab90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 >> 172.21.15.111:0/2947752071 conn(0x7f66340f6530 msgr2=0x7f66340f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 shutdown_connections 2024-02-19T22:12:48.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:48.785+0000 7f663bd47700 1 -- 172.21.15.111:0/2947752071 wait complete. 2024-02-19T22:12:48.809 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:12:47.229663+0000 mgr.smithi111.ftjzov (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-02-19T22:12:48.809 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:47.514328+0000 mon.smithi111 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:48.809 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:47.516826+0000 mon.smithi111 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:49.436 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:12:49.436 INFO:teuthology.orchestra.run.smithi111.stderr:Adding key to root@localhost authorized_keys... 2024-02-19T22:12:49.437 INFO:teuthology.orchestra.run.smithi111.stderr:Adding host smithi111... 2024-02-19T22:12:49.437 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi111 172.21.15.111 2024-02-19T22:12:49.694 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:49 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:48.784452+0000 mgr.smithi111.ftjzov (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.938+0000 7f520256a700 1 Processor -- start 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f520256a700 1 -- start start 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f520256a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f520256a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0fb6a0 con 0x7f51fc0fad90 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35138/0 (socket says 172.21.15.111:35138) 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51fbfff700 1 -- 172.21.15.111:0/3230897640 learned_addr learned my addr 172.21.15.111:0/3230897640 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:49.942 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51fbfff700 1 -- 172.21.15.111:0/3230897640 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc0fb7e0 con 0x7f51fc0fad90 2024-02-19T22:12:49.943 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51fbfff700 1 --2- 172.21.15.111:0/3230897640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f51ec00b5f0 tx=0x7f51ec006ee0).ready entity=mon.0 client_cookie=9dac01f398bad9cf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:49.943 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.939+0000 7f51faffd700 1 -- 172.21.15.111:0/3230897640 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ec00d600 con 0x7f51fc0fad90 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f51faffd700 1 -- 172.21.15.111:0/3230897640 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f51ec00dbf0 con 0x7f51fc0fad90 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f51faffd700 1 -- 172.21.15.111:0/3230897640 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ec011cb0 con 0x7f51fc0fad90 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 -- 172.21.15.111:0/3230897640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 msgr2=0x7f51fc0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 --2- 172.21.15.111:0/3230897640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f51ec00b5f0 tx=0x7f51ec006ee0).stop 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 -- 172.21.15.111:0/3230897640 shutdown_connections 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 --2- 172.21.15.111:0/3230897640 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc0fb1b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 -- 172.21.15.111:0/3230897640 >> 172.21.15.111:0/3230897640 conn(0x7f51fc0f6530 msgr2=0x7f51fc0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 -- 172.21.15.111:0/3230897640 shutdown_connections 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.940+0000 7f520256a700 1 -- 172.21.15.111:0/3230897640 wait complete. 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f520256a700 1 Processor -- start 2024-02-19T22:12:49.946 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f520256a700 1 -- start start 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f520256a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f520256a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc10b6a0 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f51fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f51fbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:35142/0 (socket says 172.21.15.111:35142) 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f51fbfff700 1 -- 172.21.15.111:0/1969870947 learned_addr learned my addr 172.21.15.111:0/1969870947 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f51fbfff700 1 -- 172.21.15.111:0/1969870947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec00b200 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.941+0000 7f51fbfff700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f51ec002f40 tx=0x7f51ec006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.942+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ec01b410 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.942+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f51ec01ba00 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.942+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc070450 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.942+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51ec01a900 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.942+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc1101a0 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc045ad0 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f51ec00f070 con 0x7f51fc0fad90 2024-02-19T22:12:49.947 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f51f8ff9700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 0x7f51e80322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:49.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f51ec041330 con 0x7f51fc0fad90 2024-02-19T22:12:49.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f51fb7fe700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 0x7f51e80322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:49.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.943+0000 7f51fb7fe700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 0x7f51e80322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f51f00096a0 tx=0x7f51f0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:49.948 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:49.947+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51ec022310 con 0x7f51fc0fad90 2024-02-19T22:12:50.119 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.117+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}) v1 -- 0x7f51fc057f10 con 0x7f51e802fe10 2024-02-19T22:12:50.715 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Added host 'smithi111' 2024-02-19T22:12:50.715 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.712+0000 7f51f8ff9700 1 -- 172.21.15.111:0/1969870947 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f51fc057f10 con 0x7f51e802fe10 2024-02-19T22:12:50.716 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.714+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 msgr2=0x7f51e80322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:50.716 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.714+0000 7f520256a700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 0x7f51e80322d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f51f00096a0 tx=0x7f51f0006b40).stop 2024-02-19T22:12:50.716 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 msgr2=0x7f51fc10fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:50.717 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f51ec002f40 tx=0x7f51ec006ee0).stop 2024-02-19T22:12:50.717 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 shutdown_connections 2024-02-19T22:12:50.717 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f51e802fe10 0x7f51e80322d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:50.718 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 --2- 172.21.15.111:0/1969870947 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f51fc0fad90 0x7f51fc10fcb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:50.718 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 >> 172.21.15.111:0/1969870947 conn(0x7f51fc0f6530 msgr2=0x7f51fc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:50.718 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 shutdown_connections 2024-02-19T22:12:50.718 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:50.715+0000 7f520256a700 1 -- 172.21.15.111:0/1969870947 wait complete. 2024-02-19T22:12:50.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:50.119317+0000 mgr.smithi111.ftjzov (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:51.313 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying mon service with default placement... 2024-02-19T22:12:51.313 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-02-19T22:12:51.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.863+0000 7ff06d130700 1 Processor -- start 2024-02-19T22:12:51.867 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.863+0000 7ff06d130700 1 -- start start 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.863+0000 7ff06d130700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.863+0000 7ff06d130700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680fef40 con 0x7ff068070430 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.863+0000 7ff066d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff066d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42938/0 (socket says 172.21.15.111:42938) 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff066d9d700 1 -- 172.21.15.111:0/2229029155 learned_addr learned my addr 172.21.15.111:0/2229029155 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff066d9d700 1 -- 172.21.15.111:0/2229029155 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680ff080 con 0x7ff068070430 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff066d9d700 1 --2- 172.21.15.111:0/2229029155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff05c00b5f0 tx=0x7ff05c006ee0).ready entity=mon.0 client_cookie=1534f9763cf81729 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff065d9b700 1 -- 172.21.15.111:0/2229029155 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff05c00d600 con 0x7ff068070430 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff065d9b700 1 -- 172.21.15.111:0/2229029155 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff05c00dbf0 con 0x7ff068070430 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.864+0000 7ff065d9b700 1 -- 172.21.15.111:0/2229029155 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff05c011cb0 con 0x7ff068070430 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- 172.21.15.111:0/2229029155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 msgr2=0x7ff0680fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 --2- 172.21.15.111:0/2229029155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7ff05c00b5f0 tx=0x7ff05c006ee0).stop 2024-02-19T22:12:51.868 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- 172.21.15.111:0/2229029155 shutdown_connections 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 --2- 172.21.15.111:0/2229029155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff0680fea50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- 172.21.15.111:0/2229029155 >> 172.21.15.111:0/2229029155 conn(0x7ff0680f6530 msgr2=0x7ff0680f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- 172.21.15.111:0/2229029155 shutdown_connections 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- 172.21.15.111:0/2229029155 wait complete. 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 Processor -- start 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- start start 2024-02-19T22:12:51.869 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:51.870 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.865+0000 7ff06d130700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06810b080 con 0x7ff068070430 2024-02-19T22:12:51.870 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff066d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff066d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42944/0 (socket says 172.21.15.111:42944) 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff066d9d700 1 -- 172.21.15.111:0/3246604074 learned_addr learned my addr 172.21.15.111:0/3246604074 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff066d9d700 1 -- 172.21.15.111:0/3246604074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c00b200 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff066d9d700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff05c00df90 tx=0x7ff05c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff05c01b410 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7ff05c01ba00 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff05c01aa30 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06810b280 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.866+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06810b710 con 0x7ff068070430 2024-02-19T22:12:51.871 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.867+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff068045ad0 con 0x7ff068070430 2024-02-19T22:12:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.868+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff05c00f070 con 0x7ff068070430 2024-02-19T22:12:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.868+0000 7ff04f7fe700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 0x7ff054032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.868+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff05c041860 con 0x7ff068070430 2024-02-19T22:12:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.868+0000 7ff06659c700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 0x7ff054032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:51.872 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.868+0000 7ff06659c700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 0x7ff054032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0580096a0 tx=0x7ff058006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:51.873 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:51.872+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff05c019cd0 con 0x7ff068070430 2024-02-19T22:12:51.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:50.713115+0000 mon.smithi111 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:51.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:12:50.713438+0000 mgr.smithi111.ftjzov (mgr.14114) 9 : cephadm [INF] Added host smithi111 2024-02-19T22:12:51.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:50.713650+0000 mon.smithi111 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:12:52.050 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.048+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7ff06810bf70 con 0x7ff05402fe70 2024-02-19T22:12:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.052+0000 7ff04f7fe700 1 -- 172.21.15.111:0/3246604074 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff06810bf70 con 0x7ff05402fe70 2024-02-19T22:12:52.054 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled mon update... 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 msgr2=0x7ff054032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 0x7ff054032330 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7ff0580096a0 tx=0x7ff058006b40).stop 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 msgr2=0x7ff06810ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7ff05c00df90 tx=0x7ff05c006ee0).stop 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 shutdown_connections 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7ff05402fe70 0x7ff054032330 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 --2- 172.21.15.111:0/3246604074 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff068070430 0x7ff06810ab90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 >> 172.21.15.111:0/3246604074 conn(0x7ff0680f6530 msgr2=0x7ff0680f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 shutdown_connections 2024-02-19T22:12:52.057 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:52.054+0000 7ff06d130700 1 -- 172.21.15.111:0/3246604074 wait complete. 2024-02-19T22:12:52.628 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying mgr service with default placement... 2024-02-19T22:12:52.628 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084bfff700 1 Processor -- start 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084bfff700 1 -- start start 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c0fed80 con 0x7f084c070430 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42952/0 (socket says 172.21.15.111:42952) 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084affd700 1 -- 172.21.15.111:0/2202852165 learned_addr learned my addr 172.21.15.111:0/2202852165 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084affd700 1 -- 172.21.15.111:0/2202852165 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0feec0 con 0x7f084c070430 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.173+0000 7f084affd700 1 --2- 172.21.15.111:0/2202852165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f083c00b5f0 tx=0x7f083c006ee0).ready entity=mon.0 client_cookie=4997096c0f7c4f4d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f0849ffb700 1 -- 172.21.15.111:0/2202852165 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c00d600 con 0x7f084c070430 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f0849ffb700 1 -- 172.21.15.111:0/2202852165 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f083c00dbf0 con 0x7f084c070430 2024-02-19T22:12:53.178 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f0849ffb700 1 -- 172.21.15.111:0/2202852165 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c011cb0 con 0x7f084c070430 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 -- 172.21.15.111:0/2202852165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 msgr2=0x7f084c0fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 --2- 172.21.15.111:0/2202852165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f083c00b5f0 tx=0x7f083c006ee0).stop 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 -- 172.21.15.111:0/2202852165 shutdown_connections 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 --2- 172.21.15.111:0/2202852165 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c0fe890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 -- 172.21.15.111:0/2202852165 >> 172.21.15.111:0/2202852165 conn(0x7f084c0f6390 msgr2=0x7f084c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 -- 172.21.15.111:0/2202852165 shutdown_connections 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.174+0000 7f084bfff700 1 -- 172.21.15.111:0/2202852165 wait complete. 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.175+0000 7f084bfff700 1 Processor -- start 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.175+0000 7f084bfff700 1 -- start start 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.175+0000 7f084bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.175+0000 7f084bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c10aec0 con 0x7f084c070430 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42960/0 (socket says 172.21.15.111:42960) 2024-02-19T22:12:53.179 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084affd700 1 -- 172.21.15.111:0/1322926241 learned_addr learned my addr 172.21.15.111:0/1322926241 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084affd700 1 -- 172.21.15.111:0/1322926241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c00b200 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084affd700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f083c00df90 tx=0x7f083c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c01b410 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c10b0c0 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f083c01ba00 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f083c01aa30 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.176+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c10b550 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.177+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f083c00f070 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.177+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c045ad0 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.177+0000 7f08337fe700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 0x7f08380322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.177+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f083c041190 con 0x7f084c070430 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.177+0000 7f084a7fc700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 0x7f08380322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:53.181 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.178+0000 7f084a7fc700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 0x7f08380322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f08400096a0 tx=0x7f0840006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:53.183 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.181+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f083c022d20 con 0x7f084c070430 2024-02-19T22:12:53.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:52.051025+0000 mgr.smithi111.ftjzov (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:53.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:12:52.051736+0000 mgr.smithi111.ftjzov (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-02-19T22:12:53.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:52.053960+0000 mon.smithi111 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:53.358 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.356+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f084c10be70 con 0x7f083802fe10 2024-02-19T22:12:53.362 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.360+0000 7f08337fe700 1 -- 172.21.15.111:0/1322926241 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f084c10be70 con 0x7f083802fe10 2024-02-19T22:12:53.362 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-02-19T22:12:53.364 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 msgr2=0x7f08380322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.364 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 0x7f08380322d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f08400096a0 tx=0x7f0840006b40).stop 2024-02-19T22:12:53.364 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 msgr2=0x7f084c10a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f083c00df90 tx=0x7f083c006ee0).stop 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 shutdown_connections 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f083802fe10 0x7f08380322d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 --2- 172.21.15.111:0/1322926241 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f084c070430 0x7f084c10a9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 >> 172.21.15.111:0/1322926241 conn(0x7f084c0f6390 msgr2=0x7f084c0f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 shutdown_connections 2024-02-19T22:12:53.365 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:53.362+0000 7f084bfff700 1 -- 172.21.15.111:0/1322926241 wait complete. 2024-02-19T22:12:53.969 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying crash service with default placement... 2024-02-19T22:12:53.969 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.442+0000 7f3cda409700 1 Processor -- start 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.442+0000 7f3cda409700 1 -- start start 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.442+0000 7f3cda409700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.442+0000 7f3cda409700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd40fc8e0 con 0x7f3cd4070390 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.442+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42972/0 (socket says 172.21.15.111:42972) 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd3fff700 1 -- 172.21.15.111:0/1224625059 learned_addr learned my addr 172.21.15.111:0/1224625059 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd3fff700 1 -- 172.21.15.111:0/1224625059 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd41013e0 con 0x7f3cd4070390 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd3fff700 1 --2- 172.21.15.111:0/1224625059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3cc800b5f0 tx=0x7f3cc8006ee0).ready entity=mon.0 client_cookie=81ce3c4f0e7c2c87 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd2ffd700 1 -- 172.21.15.111:0/1224625059 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc800d600 con 0x7f3cd4070390 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd2ffd700 1 -- 172.21.15.111:0/1224625059 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3cc800dbf0 con 0x7f3cd4070390 2024-02-19T22:12:54.447 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.443+0000 7f3cd2ffd700 1 -- 172.21.15.111:0/1224625059 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc8011cb0 con 0x7f3cd4070390 2024-02-19T22:12:54.448 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- 172.21.15.111:0/1224625059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 msgr2=0x7f3cd4100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:54.448 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 --2- 172.21.15.111:0/1224625059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f3cc800b5f0 tx=0x7f3cc8006ee0).stop 2024-02-19T22:12:54.448 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- 172.21.15.111:0/1224625059 shutdown_connections 2024-02-19T22:12:54.448 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 --2- 172.21.15.111:0/1224625059 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd4100ef0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:54.448 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- 172.21.15.111:0/1224625059 >> 172.21.15.111:0/1224625059 conn(0x7f3cd40f6530 msgr2=0x7f3cd40f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- 172.21.15.111:0/1224625059 shutdown_connections 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- 172.21.15.111:0/1224625059 wait complete. 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 Processor -- start 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.444+0000 7f3cda409700 1 -- start start 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cda409700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cda409700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd410d270 con 0x7f3cd4070390 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42984/0 (socket says 172.21.15.111:42984) 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd3fff700 1 -- 172.21.15.111:0/1619717226 learned_addr learned my addr 172.21.15.111:0/1619717226 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd3fff700 1 -- 172.21.15.111:0/1619717226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc800b200 con 0x7f3cd4070390 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd3fff700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3cc8006800 tx=0x7f3cc8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc801b570 con 0x7f3cd4070390 2024-02-19T22:12:54.449 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd410d470 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3cc801bb60 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cc801a9f0 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.445+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd410e080 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.446+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f3cc800f070 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.446+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd4059100 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.446+0000 7f3cd0ff9700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 0x7f3cc00322d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.446+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f3cc8041510 con 0x7f3cd4070390 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.446+0000 7f3cd37fe700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 0x7f3cc00322d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:54.450 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.447+0000 7f3cd37fe700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 0x7f3cc00322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3cc40096a0 tx=0x7f3cc4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:54.452 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.450+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cc8010360 con 0x7f3cd4070390 2024-02-19T22:12:54.626 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.624+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cd410e380 con 0x7f3cc002fe10 2024-02-19T22:12:54.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.628+0000 7f3cd0ff9700 1 -- 172.21.15.111:0/1619717226 <== mgr.14114 v2:172.21.15.111:6800/3561110342 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f3cd410e380 con 0x7f3cc002fe10 2024-02-19T22:12:54.630 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled crash update... 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 msgr2=0x7f3cc00322d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 0x7f3cc00322d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f3cc40096a0 tx=0x7f3cc4006b40).stop 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 msgr2=0x7f3cd410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f3cc8006800 tx=0x7f3cc8006ee0).stop 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 shutdown_connections 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7f3cc002fe10 0x7f3cc00322d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 --2- 172.21.15.111:0/1619717226 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3cd4070390 0x7f3cd410cd80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 >> 172.21.15.111:0/1619717226 conn(0x7f3cd40f6530 msgr2=0x7f3cd40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 shutdown_connections 2024-02-19T22:12:54.633 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:54.630+0000 7f3cda409700 1 -- 172.21.15.111:0/1619717226 wait complete. 2024-02-19T22:12:54.655 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:53.358733+0000 mgr.smithi111.ftjzov (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:54.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:12:53.359257+0000 mgr.smithi111.ftjzov (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-02-19T22:12:54.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:53.361558+0000 mon.smithi111 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:55.267 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling mgr prometheus module... 2024-02-19T22:12:55.267 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-02-19T22:12:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb373900700 1 Processor -- start 2024-02-19T22:12:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb373900700 1 -- start start 2024-02-19T22:12:55.763 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb373900700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb373900700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0fc8a0 con 0x7fb36c0703d0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb37169c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.752+0000 7fb37169c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42986/0 (socket says 172.21.15.111:42986) 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb37169c700 1 -- 172.21.15.111:0/4285317943 learned_addr learned my addr 172.21.15.111:0/4285317943 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb37169c700 1 -- 172.21.15.111:0/4285317943 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c1013a0 con 0x7fb36c0703d0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb37169c700 1 --2- 172.21.15.111:0/4285317943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb36800b5f0 tx=0x7fb368006ee0).ready entity=mon.0 client_cookie=10080109bad87b57 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb35bfff700 1 -- 172.21.15.111:0/4285317943 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36800d600 con 0x7fb36c0703d0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb35bfff700 1 -- 172.21.15.111:0/4285317943 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb36800dbf0 con 0x7fb36c0703d0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb35bfff700 1 -- 172.21.15.111:0/4285317943 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb368011cb0 con 0x7fb36c0703d0 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb373900700 1 -- 172.21.15.111:0/4285317943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 msgr2=0x7fb36c100eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:55.764 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb373900700 1 --2- 172.21.15.111:0/4285317943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb36800b5f0 tx=0x7fb368006ee0).stop 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb373900700 1 -- 172.21.15.111:0/4285317943 shutdown_connections 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb373900700 1 --2- 172.21.15.111:0/4285317943 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c100eb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.753+0000 7fb373900700 1 -- 172.21.15.111:0/4285317943 >> 172.21.15.111:0/4285317943 conn(0x7fb36c0f6530 msgr2=0x7fb36c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 -- 172.21.15.111:0/4285317943 shutdown_connections 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 -- 172.21.15.111:0/4285317943 wait complete. 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 Processor -- start 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 -- start start 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb373900700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c10d330 con 0x7fb36c0703d0 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb37169c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb37169c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:42996/0 (socket says 172.21.15.111:42996) 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.754+0000 7fb37169c700 1 -- 172.21.15.111:0/4260241328 learned_addr learned my addr 172.21.15.111:0/4260241328 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb37169c700 1 -- 172.21.15.111:0/4260241328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36800b200 con 0x7fb36c0703d0 2024-02-19T22:12:55.765 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb37169c700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb368006800 tx=0x7fb368006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36801b570 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb36801bb60 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c10d530 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb36801a940 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.755+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c10e140 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fb36800f070 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb359ffb700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 0x7fb354032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb36c059100 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb3680413a0 con 0x7fb36c0703d0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb370e9b700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 0x7fb354032330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.756+0000 7fb370e9b700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 0x7fb354032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:55.766 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.760+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb368016080 con 0x7fb36c0703d0 2024-02-19T22:12:55.945 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:55.943+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fb36c045ad0 con 0x7fb36c0703d0 2024-02-19T22:12:55.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:54.626758+0000 mgr.smithi111.ftjzov (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:12:55.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:12:54.627401+0000 mgr.smithi111.ftjzov (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-02-19T22:12:55.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:54.629799+0000 mon.smithi111 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.111:0/3834982855' entity='mgr.smithi111.ftjzov' 2024-02-19T22:12:56.636 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.634+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb36801bcd0 con 0x7fb36c0703d0 2024-02-19T22:12:56.637 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.636+0000 7fb359ffb700 1 -- 172.21.15.111:0/4260241328 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fb368019990 con 0x7fb36c0703d0 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 msgr2=0x7fb354032330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 0x7fb354032330 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fb35c0096a0 tx=0x7fb35c006b40).stop 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 msgr2=0x7fb36c10ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fb368006800 tx=0x7fb368006ee0).stop 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 shutdown_connections 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fb35402fe70 0x7fb354032330 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.641 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 --2- 172.21.15.111:0/4260241328 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb36c0703d0 0x7fb36c10ce40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:56.642 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 >> 172.21.15.111:0/4260241328 conn(0x7fb36c0f6530 msgr2=0x7fb36c0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:56.642 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 shutdown_connections 2024-02-19T22:12:56.642 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:56.638+0000 7fb373900700 1 -- 172.21.15.111:0/4260241328 wait complete. 2024-02-19T22:12:56.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:55.945274+0000 mon.smithi111 (mon.0) 59 : audit [INF] from='client.? 172.21.15.111:0/4260241328' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T22:12:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying prometheus service with default placement... 2024-02-19T22:12:57.283 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-02-19T22:12:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb71a2b700 1 Processor -- start 2024-02-19T22:12:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb71a2b700 1 -- start start 2024-02-19T22:12:57.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb71a2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb71a2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c0fb440 con 0x7fcb6c0fab30 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb70a29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb70a29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43024/0 (socket says 172.21.15.111:43024) 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.835+0000 7fcb70a29700 1 -- 172.21.15.111:0/4286284929 learned_addr learned my addr 172.21.15.111:0/4286284929 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.836+0000 7fcb70a29700 1 -- 172.21.15.111:0/4286284929 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0fb580 con 0x7fcb6c0fab30 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.836+0000 7fcb70a29700 1 --2- 172.21.15.111:0/4286284929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fcb6000b5f0 tx=0x7fcb60006ee0).ready entity=mon.0 client_cookie=68cd1dd2c18f0a17 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.836+0000 7fcb6b7fe700 1 -- 172.21.15.111:0/4286284929 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb6000d600 con 0x7fcb6c0fab30 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.836+0000 7fcb6b7fe700 1 -- 172.21.15.111:0/4286284929 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcb6000dbf0 con 0x7fcb6c0fab30 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.836+0000 7fcb6b7fe700 1 -- 172.21.15.111:0/4286284929 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb60011cb0 con 0x7fcb6c0fab30 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 -- 172.21.15.111:0/4286284929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 msgr2=0x7fcb6c0faf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/4286284929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fcb6000b5f0 tx=0x7fcb60006ee0).stop 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 -- 172.21.15.111:0/4286284929 shutdown_connections 2024-02-19T22:12:57.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/4286284929 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c0faf50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:12:57.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 -- 172.21.15.111:0/4286284929 >> 172.21.15.111:0/4286284929 conn(0x7fcb6c0f6390 msgr2=0x7fcb6c0f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:12:57.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 -- 172.21.15.111:0/4286284929 shutdown_connections 2024-02-19T22:12:57.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 -- 172.21.15.111:0/4286284929 wait complete. 2024-02-19T22:12:57.841 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.837+0000 7fcb71a2b700 1 Processor -- start 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb71a2b700 1 -- start start 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb71a2b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb71a2b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c070470 con 0x7fcb6c0fab30 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb70a29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb70a29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:43034/0 (socket says 172.21.15.111:43034) 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb70a29700 1 -- 172.21.15.111:0/3141082012 learned_addr learned my addr 172.21.15.111:0/3141082012 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb70a29700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6000b200 con 0x7fcb6c0fab30 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.838+0000 7fcb70a29700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcb60011ea0 tx=0x7fcb60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb6001b570 con 0x7fcb6c0fab30 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fcb6001bb60 con 0x7fcb6c0fab30 2024-02-19T22:12:57.842 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb6001a8d0 con 0x7fcb6c0fab30 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c070670 con 0x7fcb6c0fab30 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c10dd20 con 0x7fcb6c0fab30 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fcb6000f070 con 0x7fcb6c0fab30 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.839+0000 7fcb697fa700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.840+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fcb60041700 con 0x7fcb6c0fab30 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.840+0000 7fcb6bfff700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3561110342 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.840+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:12:57.843 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.840+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c045ad0 con 0x7fcb6c0fab30 2024-02-19T22:12:57.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:57.844+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb60016070 con 0x7fcb6c0fab30 2024-02-19T22:12:57.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:12:56.637251+0000 mon.smithi111 (mon.0) 60 : audit [INF] from='client.? 172.21.15.111:0/4260241328' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T22:12:57.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:12:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:12:56.637310+0000 mon.smithi111 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi111.ftjzov(active, since 14s) 2024-02-19T22:12:58.043 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.041+0000 7fcb6bfff700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3561110342 2024-02-19T22:12:58.043 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.041+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:12:58.063 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.062+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb6c0fca70 con 0x7fcb5002fe70 2024-02-19T22:12:58.444 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.442+0000 7fcb6bfff700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3561110342 2024-02-19T22:12:58.444 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:58.442+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:12:59.245 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.243+0000 7fcb6bfff700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3561110342 2024-02-19T22:12:59.245 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:12:59.243+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:13:00.846 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.844+0000 7fcb6bfff700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/3561110342 2024-02-19T22:13:00.847 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:00.844+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-02-19T22:13:02.153 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:02.151+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fcb6001bcd0 con 0x7fcb6c0fab30 2024-02-19T22:13:02.153 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:02.151+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 msgr2=0x7fcb50032330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:02.153 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:02.151+0000 7fcb697fa700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:02.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:01.595172+0000 mon.smithi111 (mon.0) 62 : cluster [INF] Active manager daemon smithi111.ftjzov restarted 2024-02-19T22:13:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:01.595602+0000 mon.smithi111 (mon.0) 63 : cluster [INF] Activating manager daemon smithi111.ftjzov 2024-02-19T22:13:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:01.597185+0000 mon.smithi111 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-02-19T22:13:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.155+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fcb600436a0 con 0x7fcb6c0fab30 2024-02-19T22:13:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.155+0000 7fcb697fa700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 0x7fcb500360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.155+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 --> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb6c0fca70 con 0x7fcb50033ce0 2024-02-19T22:13:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.155+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 0x7fcb500360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:03.158 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.156+0000 7fcb6bfff700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 0x7fcb500360d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fcb580036d0 tx=0x7fcb58006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:03.162 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.161+0000 7fcb697fa700 1 -- 172.21.15.111:0/3141082012 <== mgr.14142 v2:172.21.15.111:6800/959358532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fcb6c0fca70 con 0x7fcb50033ce0 2024-02-19T22:13:03.163 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-02-19T22:13:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.162+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 msgr2=0x7fcb500360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.162+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 0x7fcb500360d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fcb580036d0 tx=0x7fcb58006cb0).stop 2024-02-19T22:13:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.162+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 msgr2=0x7fcb6c10d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:03.165 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.162+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcb60011ea0 tx=0x7fcb60006ee0).stop 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 shutdown_connections 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fcb50033ce0 0x7fcb500360d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcb6c0fab30 0x7fcb6c10d670 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 --2- 172.21.15.111:0/3141082012 >> [v2:172.21.15.111:6800/3561110342,v1:172.21.15.111:6801/3561110342] conn(0x7fcb5002fe70 0x7fcb50032330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 >> 172.21.15.111:0/3141082012 conn(0x7fcb6c0f6390 msgr2=0x7fcb6c0f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 shutdown_connections 2024-02-19T22:13:03.166 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:03.163+0000 7fcb71a2b700 1 -- 172.21.15.111:0/3141082012 wait complete. 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:02.153655+0000 mon.smithi111 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi111.ftjzov(active, starting, since 0.558132s) 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.154601+0000 mon.smithi111 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.155177+0000 mon.smithi111 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mgr metadata", "who": "smithi111.ftjzov", "id": "smithi111.ftjzov"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.155927+0000 mon.smithi111 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.156095+0000 mon.smithi111 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.156244+0000 mon.smithi111 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:02.169943+0000 mon.smithi111 (mon.0) 71 : cluster [INF] Manager daemon smithi111.ftjzov is now available 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.268159+0000 mon.smithi111 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.269658+0000 mon.smithi111 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.283567+0000 mon.smithi111 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:03.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:02.286192+0000 mon.smithi111 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:03.709 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying grafana service with default placement... 2024-02-19T22:13:03.709 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-02-19T22:13:04.238 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.226+0000 7fac33d01700 1 Processor -- start 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.226+0000 7fac33d01700 1 -- start start 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.226+0000 7fac33d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.226+0000 7fac33d01700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0fef40 con 0x7fac2c070430 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac31a9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac31a9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58784/0 (socket says 172.21.15.111:58784) 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac31a9d700 1 -- 172.21.15.111:0/3030862852 learned_addr learned my addr 172.21.15.111:0/3030862852 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac31a9d700 1 -- 172.21.15.111:0/3030862852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c0ff080 con 0x7fac2c070430 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac31a9d700 1 --2- 172.21.15.111:0/3030862852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fac1c00b5f0 tx=0x7fac1c006ee0).ready entity=mon.0 client_cookie=782e84898b466fc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac30a9b700 1 -- 172.21.15.111:0/3030862852 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c00d600 con 0x7fac2c070430 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac30a9b700 1 -- 172.21.15.111:0/3030862852 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fac1c00dbf0 con 0x7fac2c070430 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.227+0000 7fac30a9b700 1 -- 172.21.15.111:0/3030862852 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c011cb0 con 0x7fac2c070430 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- 172.21.15.111:0/3030862852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 msgr2=0x7fac2c0fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.239 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 --2- 172.21.15.111:0/3030862852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fac1c00b5f0 tx=0x7fac1c006ee0).stop 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- 172.21.15.111:0/3030862852 shutdown_connections 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 --2- 172.21.15.111:0/3030862852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fea50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- 172.21.15.111:0/3030862852 >> 172.21.15.111:0/3030862852 conn(0x7fac2c0f6530 msgr2=0x7fac2c0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- 172.21.15.111:0/3030862852 shutdown_connections 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- 172.21.15.111:0/3030862852 wait complete. 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 Processor -- start 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- start start 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac33d01700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c0fa950 con 0x7fac2c070430 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac31a9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac31a9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58796/0 (socket says 172.21.15.111:58796) 2024-02-19T22:13:04.240 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.228+0000 7fac31a9d700 1 -- 172.21.15.111:0/2798615000 learned_addr learned my addr 172.21.15.111:0/2798615000 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac31a9d700 1 -- 172.21.15.111:0/2798615000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1c00b200 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac31a9d700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fac1c006800 tx=0x7fac1c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c01b570 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fac1c01bb60 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c0faaf0 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac1c01a9f0 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.229+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c0faf80 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fac1c00f070 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c045ad0 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac1a7fc700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 0x7fac140323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fac1c0415e0 con 0x7fac2c070430 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac3129c700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 0x7fac140323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.230+0000 7fac3129c700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 0x7fac140323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fac280096a0 tx=0x7fac28006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:04.241 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.234+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac1c019c90 con 0x7fac2c070430 2024-02-19T22:13:04.411 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.409+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 --> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fac2c0fff40 con 0x7fac1402ff20 2024-02-19T22:13:04.417 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.415+0000 7fac1a7fc700 1 -- 172.21.15.111:0/2798615000 <== mgr.14142 v2:172.21.15.111:6800/959358532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fac2c0fff40 con 0x7fac1402ff20 2024-02-19T22:13:04.417 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 msgr2=0x7fac140323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 0x7fac140323e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fac280096a0 tx=0x7fac28006b40).stop 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 msgr2=0x7fac2c0fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fac1c006800 tx=0x7fac1c006ee0).stop 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 shutdown_connections 2024-02-19T22:13:04.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fac1402ff20 0x7fac140323e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.421 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 --2- 172.21.15.111:0/2798615000 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fac2c070430 0x7fac2c0fc4d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:04.421 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 >> 172.21.15.111:0/2798615000 conn(0x7fac2c0f6530 msgr2=0x7fac2c0f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:04.421 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 shutdown_connections 2024-02-19T22:13:04.421 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:04.417+0000 7fac33d01700 1 -- 172.21.15.111:0/2798615000 wait complete. 2024-02-19T22:13:04.440 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:03.158314+0000 mon.smithi111 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi111.ftjzov(active, since 1.56279s) 2024-02-19T22:13:04.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:03.159337+0000 mgr.smithi111.ftjzov (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-02-19T22:13:04.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:03.162078+0000 mon.smithi111 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying node-exporter service with default placement... 2024-02-19T22:13:05.049 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-02-19T22:13:05.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:04.164141+0000 mon.smithi111 (mon.0) 78 : cluster [DBG] mgrmap e12: smithi111.ftjzov(active, since 2s) 2024-02-19T22:13:05.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:04.411284+0000 mgr.smithi111.ftjzov (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:05.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:04.411801+0000 mgr.smithi111.ftjzov (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-02-19T22:13:05.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:04.416603+0000 mon.smithi111 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:05.574 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.565+0000 7fb72b792700 1 Processor -- start 2024-02-19T22:13:05.574 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.565+0000 7fb72b792700 1 -- start start 2024-02-19T22:13:05.574 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.565+0000 7fb72b792700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:05.575 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.565+0000 7fb72b792700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7240fc8e0 con 0x7fb724070390 2024-02-19T22:13:05.575 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.565+0000 7fb72952e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:05.575 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb72952e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58804/0 (socket says 172.21.15.111:58804) 2024-02-19T22:13:05.575 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb72952e700 1 -- 172.21.15.111:0/768260825 learned_addr learned my addr 172.21.15.111:0/768260825 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb72952e700 1 -- 172.21.15.111:0/768260825 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7241013e0 con 0x7fb724070390 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb72952e700 1 --2- 172.21.15.111:0/768260825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb71400b5f0 tx=0x7fb714006ee0).ready entity=mon.0 client_cookie=c0ddd65944d2d70c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb713fff700 1 -- 172.21.15.111:0/768260825 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71400d600 con 0x7fb724070390 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb713fff700 1 -- 172.21.15.111:0/768260825 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb71400dbf0 con 0x7fb724070390 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.566+0000 7fb713fff700 1 -- 172.21.15.111:0/768260825 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb714011cb0 con 0x7fb724070390 2024-02-19T22:13:05.576 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- 172.21.15.111:0/768260825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 msgr2=0x7fb724100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 --2- 172.21.15.111:0/768260825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fb71400b5f0 tx=0x7fb714006ee0).stop 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- 172.21.15.111:0/768260825 shutdown_connections 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 --2- 172.21.15.111:0/768260825 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb724100ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- 172.21.15.111:0/768260825 >> 172.21.15.111:0/768260825 conn(0x7fb7240f6530 msgr2=0x7fb7240f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- 172.21.15.111:0/768260825 shutdown_connections 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- 172.21.15.111:0/768260825 wait complete. 2024-02-19T22:13:05.577 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 Processor -- start 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.567+0000 7fb72b792700 1 -- start start 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72b792700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72b792700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72410d270 con 0x7fb724070390 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72952e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72952e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58814/0 (socket says 172.21.15.111:58814) 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72952e700 1 -- 172.21.15.111:0/818358456 learned_addr learned my addr 172.21.15.111:0/818358456 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:05.578 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72952e700 1 -- 172.21.15.111:0/818358456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71400b200 con 0x7fb724070390 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72952e700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb714006800 tx=0x7fb714006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71401b570 con 0x7fb724070390 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fb71401bb60 con 0x7fb724070390 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72410d470 con 0x7fb724070390 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb71401a9f0 con 0x7fb724070390 2024-02-19T22:13:05.579 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.568+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72410e080 con 0x7fb724070390 2024-02-19T22:13:05.580 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.569+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fb71400f070 con 0x7fb724070390 2024-02-19T22:13:05.580 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.569+0000 7fb711ffb700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 0x7fb70c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:05.580 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.569+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb724059100 con 0x7fb724070390 2024-02-19T22:13:05.580 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.569+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fb714041580 con 0x7fb724070390 2024-02-19T22:13:05.581 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.570+0000 7fb728d2d700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 0x7fb70c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:05.581 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.570+0000 7fb728d2d700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 0x7fb70c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb7200096a0 tx=0x7fb720006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:05.582 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.574+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb714016020 con 0x7fb724070390 2024-02-19T22:13:05.751 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.749+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 --> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fb72410e380 con 0x7fb70c02ff70 2024-02-19T22:13:05.755 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.753+0000 7fb711ffb700 1 -- 172.21.15.111:0/818358456 <== mgr.14142 v2:172.21.15.111:6800/959358532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb72410e380 con 0x7fb70c02ff70 2024-02-19T22:13:05.755 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 msgr2=0x7fb70c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 0x7fb70c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fb7200096a0 tx=0x7fb720006b40).stop 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 msgr2=0x7fb72410cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb714006800 tx=0x7fb714006ee0).stop 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 shutdown_connections 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fb70c02ff70 0x7fb70c032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 --2- 172.21.15.111:0/818358456 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb724070390 0x7fb72410cd80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 >> 172.21.15.111:0/818358456 conn(0x7fb7240f6530 msgr2=0x7fb7240f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 shutdown_connections 2024-02-19T22:13:05.758 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:05.755+0000 7fb72b792700 1 -- 172.21.15.111:0/818358456 wait complete. 2024-02-19T22:13:06.330 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying alertmanager service with default placement... 2024-02-19T22:13:06.330 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-02-19T22:13:06.903 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.868+0000 7f60d04dd700 1 Processor -- start 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.868+0000 7f60d04dd700 1 -- start start 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.868+0000 7f60d04dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.868+0000 7f60d04dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c80fef80 con 0x7f60c80703f0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.868+0000 7f60ce279700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60ce279700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58820/0 (socket says 172.21.15.111:58820) 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60ce279700 1 -- 172.21.15.111:0/2043322624 learned_addr learned my addr 172.21.15.111:0/2043322624 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60ce279700 1 -- 172.21.15.111:0/2043322624 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c80ff0c0 con 0x7f60c80703f0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60ce279700 1 --2- 172.21.15.111:0/2043322624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f60b800b5f0 tx=0x7f60b8006ee0).ready entity=mon.0 client_cookie=44d01dd2636e6f2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60cd277700 1 -- 172.21.15.111:0/2043322624 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60b800d600 con 0x7f60c80703f0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60cd277700 1 -- 172.21.15.111:0/2043322624 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f60b800dbf0 con 0x7f60c80703f0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60cd277700 1 -- 172.21.15.111:0/2043322624 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60b8011ce0 con 0x7f60c80703f0 2024-02-19T22:13:06.904 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60d04dd700 1 -- 172.21.15.111:0/2043322624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 msgr2=0x7f60c80fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.869+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2043322624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f60b800b5f0 tx=0x7f60b8006ee0).stop 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- 172.21.15.111:0/2043322624 shutdown_connections 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2043322624 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c80fea90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- 172.21.15.111:0/2043322624 >> 172.21.15.111:0/2043322624 conn(0x7f60c80f6530 msgr2=0x7f60c80f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- 172.21.15.111:0/2043322624 shutdown_connections 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- 172.21.15.111:0/2043322624 wait complete. 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 Processor -- start 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- start start 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60d04dd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60c810b040 con 0x7f60c80703f0 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60ce279700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60ce279700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58828/0 (socket says 172.21.15.111:58828) 2024-02-19T22:13:06.907 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.870+0000 7f60ce279700 1 -- 172.21.15.111:0/2838631721 learned_addr learned my addr 172.21.15.111:0/2838631721 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60ce279700 1 -- 172.21.15.111:0/2838631721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b800b200 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60ce279700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f60b8011ed0 tx=0x7f60b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60b801b570 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f60b801bb60 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c810b240 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60b801aad0 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.871+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c810b6d0 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f60b800f070 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60b6ffd700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 0x7f60b00323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c8045ad0 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f60b8041550 con 0x7f60c80703f0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60cda78700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 0x7f60b00323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:06.908 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.872+0000 7f60cda78700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 0x7f60b00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f60c40096a0 tx=0x7f60c4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:06.909 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:06.876+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60b8016070 con 0x7f60c80703f0 2024-02-19T22:13:06.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:05.752006+0000 mgr.smithi111.ftjzov (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:06.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:05.752662+0000 mgr.smithi111.ftjzov (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-02-19T22:13:06.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:05.754930+0000 mon.smithi111 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:07.054 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.052+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 --> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f60c810bff0 con 0x7f60b002ff20 2024-02-19T22:13:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.056+0000 7f60b6ffd700 1 -- 172.21.15.111:0/2838631721 <== mgr.14142 v2:172.21.15.111:6800/959358532 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f60c810bff0 con 0x7f60b002ff20 2024-02-19T22:13:07.058 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-02-19T22:13:07.060 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 msgr2=0x7f60b00323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 0x7f60b00323e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f60c40096a0 tx=0x7f60c4006b40).stop 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 msgr2=0x7f60c810ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f60b8011ed0 tx=0x7f60b8006ee0).stop 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 shutdown_connections 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f60b002ff20 0x7f60b00323e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 --2- 172.21.15.111:0/2838631721 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f60c80703f0 0x7f60c810ab50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 >> 172.21.15.111:0/2838631721 conn(0x7f60c80f6530 msgr2=0x7f60c80f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 shutdown_connections 2024-02-19T22:13:07.061 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:07.058+0000 7f60d04dd700 1 -- 172.21.15.111:0/2838631721 wait complete. 2024-02-19T22:13:07.604 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-02-19T22:13:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d359e700 1 Processor -- start 2024-02-19T22:13:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d359e700 1 -- start start 2024-02-19T22:13:08.123 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d40fed80 con 0x7f59d4070430 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58842/0 (socket says 172.21.15.111:58842) 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d259c700 1 -- 172.21.15.111:0/3571294450 learned_addr learned my addr 172.21.15.111:0/3571294450 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.119+0000 7f59d259c700 1 -- 172.21.15.111:0/3571294450 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d40feec0 con 0x7f59d4070430 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d259c700 1 --2- 172.21.15.111:0/3571294450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f59bc00b5f0 tx=0x7f59bc006ee0).ready entity=mon.0 client_cookie=fae9aa8119cf6172 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d159a700 1 -- 172.21.15.111:0/3571294450 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59bc00d600 con 0x7f59d4070430 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d159a700 1 -- 172.21.15.111:0/3571294450 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f59bc00dbf0 con 0x7f59d4070430 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d159a700 1 -- 172.21.15.111:0/3571294450 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59bc011cb0 con 0x7f59d4070430 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 -- 172.21.15.111:0/3571294450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 msgr2=0x7f59d40fe890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 --2- 172.21.15.111:0/3571294450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f59bc00b5f0 tx=0x7f59bc006ee0).stop 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 -- 172.21.15.111:0/3571294450 shutdown_connections 2024-02-19T22:13:08.124 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 --2- 172.21.15.111:0/3571294450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d40fe890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 -- 172.21.15.111:0/3571294450 >> 172.21.15.111:0/3571294450 conn(0x7f59d40f6390 msgr2=0x7f59d40f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 -- 172.21.15.111:0/3571294450 shutdown_connections 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.120+0000 7f59d359e700 1 -- 172.21.15.111:0/3571294450 wait complete. 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d359e700 1 Processor -- start 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d359e700 1 -- start start 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d410aec0 con 0x7f59d4070430 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58846/0 (socket says 172.21.15.111:58846) 2024-02-19T22:13:08.125 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.121+0000 7f59d259c700 1 -- 172.21.15.111:0/2649834707 learned_addr learned my addr 172.21.15.111:0/2649834707 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59d259c700 1 -- 172.21.15.111:0/2649834707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc00b200 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59d259c700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f59bc00df90 tx=0x7f59bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59bc01b410 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f59bc01ba00 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d410b0c0 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59bc01aa30 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.122+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d410b550 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.123+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f59bc00f070 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.123+0000 7f59caffd700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 0x7f59b4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.123+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d4045ad0 con 0x7f59d4070430 2024-02-19T22:13:08.127 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.123+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f59bc0412f0 con 0x7f59d4070430 2024-02-19T22:13:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.123+0000 7f59d1d9b700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 0x7f59b4032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:08.128 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.124+0000 7f59d1d9b700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 0x7f59b4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:08.129 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.127+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59bc022af0 con 0x7f59d4070430 2024-02-19T22:13:08.208 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:07.054757+0000 mgr.smithi111.ftjzov (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:08.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:07.055291+0000 mgr.smithi111.ftjzov (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-02-19T22:13:08.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:07.057462+0000 mon.smithi111 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.111:0/3634016999' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:08.279 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.277+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f59d40ffc00 con 0x7f59d4070430 2024-02-19T22:13:08.281 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.280+0000 7f59caffd700 1 -- 172.21.15.111:0/2649834707 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f59bc022310 con 0x7f59d4070430 2024-02-19T22:13:08.284 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 msgr2=0x7f59b4032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 0x7f59b4032430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f59c40096a0 tx=0x7f59c4006b40).stop 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 msgr2=0x7f59d410a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f59bc00df90 tx=0x7f59bc006ee0).stop 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 shutdown_connections 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f59b402ff70 0x7f59b4032430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 --2- 172.21.15.111:0/2649834707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59d4070430 0x7f59d410a9d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.281+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 >> 172.21.15.111:0/2649834707 conn(0x7f59d40f6390 msgr2=0x7f59d40f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.282+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 shutdown_connections 2024-02-19T22:13:08.285 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:08.282+0000 7f59d359e700 1 -- 172.21.15.111:0/2649834707 wait complete. 2024-02-19T22:13:08.869 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-02-19T22:13:09.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd125413700 1 Processor -- start 2024-02-19T22:13:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd125413700 1 -- start start 2024-02-19T22:13:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd125413700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd125413700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120072960 con 0x7fd120072540 2024-02-19T22:13:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd11effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:09.414 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd11effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58858/0 (socket says 172.21.15.111:58858) 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd11effd700 1 -- 172.21.15.111:0/2687428407 learned_addr learned my addr 172.21.15.111:0/2687428407 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.380+0000 7fd11effd700 1 -- 172.21.15.111:0/2687428407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120070fd0 con 0x7fd120072540 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd11effd700 1 --2- 172.21.15.111:0/2687428407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd11000b5f0 tx=0x7fd110006ee0).ready entity=mon.0 client_cookie=c5c8ca9cebbc43c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd11dffb700 1 -- 172.21.15.111:0/2687428407 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11000d600 con 0x7fd120072540 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd11dffb700 1 -- 172.21.15.111:0/2687428407 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd11000dbf0 con 0x7fd120072540 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd11dffb700 1 -- 172.21.15.111:0/2687428407 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd110011cb0 con 0x7fd120072540 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 -- 172.21.15.111:0/2687428407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 msgr2=0x7fd120070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 --2- 172.21.15.111:0/2687428407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd11000b5f0 tx=0x7fd110006ee0).stop 2024-02-19T22:13:09.415 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 -- 172.21.15.111:0/2687428407 shutdown_connections 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 --2- 172.21.15.111:0/2687428407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd120070ae0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 -- 172.21.15.111:0/2687428407 >> 172.21.15.111:0/2687428407 conn(0x7fd1200f76a0 msgr2=0x7fd1200f9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 -- 172.21.15.111:0/2687428407 shutdown_connections 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.381+0000 7fd125413700 1 -- 172.21.15.111:0/2687428407 wait complete. 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd125413700 1 Processor -- start 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd125413700 1 -- start start 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd125413700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd125413700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12010d2f0 con 0x7fd120072540 2024-02-19T22:13:09.418 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd11effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd11effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:58874/0 (socket says 172.21.15.111:58874) 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd11effd700 1 -- 172.21.15.111:0/2224065813 learned_addr learned my addr 172.21.15.111:0/2224065813 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.382+0000 7fd11effd700 1 -- 172.21.15.111:0/2224065813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11000b200 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd11effd700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd110006800 tx=0x7fd110006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11001b570 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fd11001bb60 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12010d4f0 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd11001a9f0 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12010e100 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.383+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fd11000f070 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.384+0000 7fd1077fe700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 0x7fd10c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.384+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd120059100 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.384+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd110041630 con 0x7fd120072540 2024-02-19T22:13:09.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.384+0000 7fd11e7fc700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 0x7fd10c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.384+0000 7fd11e7fc700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 0x7fd10c032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1140096a0 tx=0x7fd114006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:09.420 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.388+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd110016070 con 0x7fd120072540 2024-02-19T22:13:09.542 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.540+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd120045ad0 con 0x7fd120072540 2024-02-19T22:13:09.544 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.543+0000 7fd1077fe700 1 -- 172.21.15.111:0/2224065813 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd110019cd0 con 0x7fd120072540 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 msgr2=0x7fd10c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 0x7fd10c032430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fd1140096a0 tx=0x7fd114006b40).stop 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 msgr2=0x7fd12010ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd110006800 tx=0x7fd110006ee0).stop 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 shutdown_connections 2024-02-19T22:13:09.547 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fd10c02ff70 0x7fd10c032430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:09.548 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 --2- 172.21.15.111:0/2224065813 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd120072540 0x7fd12010ce00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:09.548 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.544+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 >> 172.21.15.111:0/2224065813 conn(0x7fd1200f76a0 msgr2=0x7fd1200f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:09.548 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.545+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 shutdown_connections 2024-02-19T22:13:09.548 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:09.545+0000 7fd125413700 1 -- 172.21.15.111:0/2224065813 wait complete. 2024-02-19T22:13:09.578 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:09 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:08.281411+0000 mon.smithi111 (mon.0) 82 : audit [INF] from='client.? 172.21.15.111:0/2649834707' entity='client.admin' 2024-02-19T22:13:10.203 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling the dashboard module... 2024-02-19T22:13:10.203 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-02-19T22:13:10.958 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:09.544225+0000 mon.smithi111 (mon.0) 83 : audit [INF] from='client.? 172.21.15.111:0/2224065813' entity='client.admin' 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.871+0000 7f408ee40700 1 Processor -- start 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.871+0000 7f408ee40700 1 -- start start 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.871+0000 7f408ee40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.871+0000 7f408ee40700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40880fb6a0 con 0x7f40880fad90 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.871+0000 7f408cbdc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f408cbdc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39692/0 (socket says 172.21.15.111:39692) 2024-02-19T22:13:10.989 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f408cbdc700 1 -- 172.21.15.111:0/179859087 learned_addr learned my addr 172.21.15.111:0/179859087 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f408cbdc700 1 -- 172.21.15.111:0/179859087 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40880fb7e0 con 0x7f40880fad90 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f408cbdc700 1 --2- 172.21.15.111:0/179859087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f407c00b5f0 tx=0x7f407c006ee0).ready entity=mon.0 client_cookie=2e6f5b03e7c40e16 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f40877fe700 1 -- 172.21.15.111:0/179859087 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f407c00d600 con 0x7f40880fad90 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f40877fe700 1 -- 172.21.15.111:0/179859087 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f407c00dbf0 con 0x7f40880fad90 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.872+0000 7f40877fe700 1 -- 172.21.15.111:0/179859087 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f407c011cb0 con 0x7f40880fad90 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 -- 172.21.15.111:0/179859087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 msgr2=0x7f40880fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 --2- 172.21.15.111:0/179859087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f407c00b5f0 tx=0x7f407c006ee0).stop 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 -- 172.21.15.111:0/179859087 shutdown_connections 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 --2- 172.21.15.111:0/179859087 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f40880fb1b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 -- 172.21.15.111:0/179859087 >> 172.21.15.111:0/179859087 conn(0x7f40880f6530 msgr2=0x7f40880f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 -- 172.21.15.111:0/179859087 shutdown_connections 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.873+0000 7f408ee40700 1 -- 172.21.15.111:0/179859087 wait complete. 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408ee40700 1 Processor -- start 2024-02-19T22:13:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408ee40700 1 -- start start 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408ee40700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408ee40700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4088070450 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408cbdc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408cbdc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39698/0 (socket says 172.21.15.111:39698) 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408cbdc700 1 -- 172.21.15.111:0/3348937025 learned_addr learned my addr 172.21.15.111:0/3348937025 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408cbdc700 1 -- 172.21.15.111:0/3348937025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c00b200 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.874+0000 7f408cbdc700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f407c011ea0 tx=0x7f407c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.875+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f407c01b570 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.875+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4088070650 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.875+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f407c01bb60 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.875+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f407c01a8d0 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.875+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408810ded0 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f407c00f070 con 0x7f40880fad90 2024-02-19T22:13:10.991 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4088045ad0 con 0x7f40880fad90 2024-02-19T22:13:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f40857fa700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 0x7f406c0323e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f407c041630 con 0x7f40880fad90 2024-02-19T22:13:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f4087fff700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 0x7f406c0323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.876+0000 7f4087fff700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 0x7f406c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f40740096a0 tx=0x7f4074006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:10.992 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:10.880+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f407c022d20 con 0x7f40880fad90 2024-02-19T22:13:11.077 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.075+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f4088027a00 con 0x7f40880fad90 2024-02-19T22:13:11.548 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.546+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f407c025980 con 0x7f40880fad90 2024-02-19T22:13:11.549 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.547+0000 7f40857fa700 1 -- 172.21.15.111:0/3348937025 <== mon.0 v2:172.21.15.111:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f407c0107d0 con 0x7f40880fad90 2024-02-19T22:13:11.551 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 msgr2=0x7f406c0323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 0x7f406c0323e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f40740096a0 tx=0x7f4074006b40).stop 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 msgr2=0x7f408810d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f407c011ea0 tx=0x7f407c006ee0).stop 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 shutdown_connections 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f406c02ff20 0x7f406c0323e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 --2- 172.21.15.111:0/3348937025 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f40880fad90 0x7f408810d820 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:11.552 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.549+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 >> 172.21.15.111:0/3348937025 conn(0x7f40880f6530 msgr2=0x7f40880f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:11.553 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.550+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 shutdown_connections 2024-02-19T22:13:11.553 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:11.550+0000 7f408ee40700 1 -- 172.21.15.111:0/3348937025 wait complete. 2024-02-19T22:13:11.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:11.077410+0000 mon.smithi111 (mon.0) 84 : audit [INF] from='client.? 172.21.15.111:0/3348937025' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T22:13:12.351 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-02-19T22:13:12.631 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:11.548795+0000 mon.smithi111 (mon.0) 85 : audit [INF] from='client.? 172.21.15.111:0/3348937025' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T22:13:12.631 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:11.548845+0000 mon.smithi111 (mon.0) 86 : cluster [DBG] mgrmap e13: smithi111.ftjzov(active, since 9s) 2024-02-19T22:13:12.952 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.948+0000 7f74d6c7d700 1 Processor -- start 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.948+0000 7f74d6c7d700 1 -- start start 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.948+0000 7f74d6c7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.948+0000 7f74d6c7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d00fef40 con 0x7f74d0070430 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74d4a19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74d4a19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39722/0 (socket says 172.21.15.111:39722) 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74d4a19700 1 -- 172.21.15.111:0/3047190438 learned_addr learned my addr 172.21.15.111:0/3047190438 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74d4a19700 1 -- 172.21.15.111:0/3047190438 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d00ff080 con 0x7f74d0070430 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74d4a19700 1 --2- 172.21.15.111:0/3047190438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f74c400b5f0 tx=0x7f74c4006ee0).ready entity=mon.0 client_cookie=78f30bec555d3c2c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74cf7fe700 1 -- 172.21.15.111:0/3047190438 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c400d600 con 0x7f74d0070430 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74cf7fe700 1 -- 172.21.15.111:0/3047190438 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f74c400dbf0 con 0x7f74d0070430 2024-02-19T22:13:12.953 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.949+0000 7f74cf7fe700 1 -- 172.21.15.111:0/3047190438 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c4011cb0 con 0x7f74d0070430 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 -- 172.21.15.111:0/3047190438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 msgr2=0x7f74d00fea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/3047190438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f74c400b5f0 tx=0x7f74c4006ee0).stop 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 -- 172.21.15.111:0/3047190438 shutdown_connections 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/3047190438 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fea50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 -- 172.21.15.111:0/3047190438 >> 172.21.15.111:0/3047190438 conn(0x7f74d00f6530 msgr2=0x7f74d00f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 -- 172.21.15.111:0/3047190438 shutdown_connections 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.950+0000 7f74d6c7d700 1 -- 172.21.15.111:0/3047190438 wait complete. 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d6c7d700 1 Processor -- start 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d6c7d700 1 -- start start 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d6c7d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d6c7d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d00fa950 con 0x7f74d0070430 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d4a19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:12.954 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d4a19700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39730/0 (socket says 172.21.15.111:39730) 2024-02-19T22:13:12.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d4a19700 1 -- 172.21.15.111:0/2541340273 learned_addr learned my addr 172.21.15.111:0/2541340273 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:12.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d4a19700 1 -- 172.21.15.111:0/2541340273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c400b200 con 0x7f74d0070430 2024-02-19T22:13:12.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.951+0000 7f74d4a19700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f74c400df90 tx=0x7f74c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:12.955 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.952+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c401b410 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.952+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d00faaf0 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.952+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f74c401ba00 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.952+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74c401aa30 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.952+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d00faf80 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f74c400f070 con 0x7f74d0070430 2024-02-19T22:13:12.956 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74cd7fa700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 0x7f74b4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:12.957 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f74c4041360 con 0x7f74d0070430 2024-02-19T22:13:12.957 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74cffff700 1 -- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 msgr2=0x7f74b4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:12.957 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74cffff700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 0x7f74b4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:12.957 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.953+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d0059100 con 0x7f74d0070430 2024-02-19T22:13:12.958 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:12.957+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74c4010360 con 0x7f74d0070430 2024-02-19T22:13:13.153 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.152+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f74d0104a80 con 0x7f74d0070430 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.152+0000 7f74cd7fa700 1 -- 172.21.15.111:0/2541340273 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f74c4022ac0 con 0x7f74d0070430 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 13, 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "active_name": "smithi111.ftjzov", 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standby": 0 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.154+0000 7f74cffff700 1 -- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 msgr2=0x7f74b4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:13.154 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.154+0000 7f74cffff700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 0x7f74b4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:13:13.156 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 msgr2=0x7f74b4032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:13.156 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 0x7f74b4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 msgr2=0x7f74d00fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f74c400df90 tx=0x7f74c4006ee0).stop 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 shutdown_connections 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7f74b402ff70 0x7f74b4032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 --2- 172.21.15.111:0/2541340273 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74d0070430 0x7f74d00fc4d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 >> 172.21.15.111:0/2541340273 conn(0x7f74d00f6530 msgr2=0x7f74d00f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 shutdown_connections 2024-02-19T22:13:13.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:13.155+0000 7f74d6c7d700 1 -- 172.21.15.111:0/2541340273 wait complete. 2024-02-19T22:13:13.708 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:13 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:13.152694+0000 mon.smithi111 (mon.0) 87 : audit [DBG] from='client.? 172.21.15.111:0/2541340273' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T22:13:13.782 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for the mgr to restart... 2024-02-19T22:13:13.782 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr epoch 13... 2024-02-19T22:13:13.782 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.283+0000 7fbbf3834700 1 Processor -- start 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.283+0000 7fbbf3834700 1 -- start start 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf3834700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf3834700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec0fb6a0 con 0x7fbbec0fad90 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf15d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf15d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39746/0 (socket says 172.21.15.111:39746) 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf15d0700 1 -- 172.21.15.111:0/280992186 learned_addr learned my addr 172.21.15.111:0/280992186 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf15d0700 1 -- 172.21.15.111:0/280992186 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec0fb7e0 con 0x7fbbec0fad90 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbf15d0700 1 --2- 172.21.15.111:0/280992186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbbdc00b5f0 tx=0x7fbbdc006ee0).ready entity=mon.0 client_cookie=6eb283e5591a4ab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbdbfff700 1 -- 172.21.15.111:0/280992186 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbdc00d600 con 0x7fbbec0fad90 2024-02-19T22:13:14.288 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.284+0000 7fbbdbfff700 1 -- 172.21.15.111:0/280992186 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbbdc00dbf0 con 0x7fbbec0fad90 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbdbfff700 1 -- 172.21.15.111:0/280992186 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbdc011cb0 con 0x7fbbec0fad90 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 -- 172.21.15.111:0/280992186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 msgr2=0x7fbbec0fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 --2- 172.21.15.111:0/280992186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbbdc00b5f0 tx=0x7fbbdc006ee0).stop 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 -- 172.21.15.111:0/280992186 shutdown_connections 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 --2- 172.21.15.111:0/280992186 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec0fb1b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 -- 172.21.15.111:0/280992186 >> 172.21.15.111:0/280992186 conn(0x7fbbec0f6530 msgr2=0x7fbbec0f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 -- 172.21.15.111:0/280992186 shutdown_connections 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.285+0000 7fbbf3834700 1 -- 172.21.15.111:0/280992186 wait complete. 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf3834700 1 Processor -- start 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf3834700 1 -- start start 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf3834700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf3834700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec0ff4c0 con 0x7fbbec0fad90 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf15d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:14.289 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf15d0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39760/0 (socket says 172.21.15.111:39760) 2024-02-19T22:13:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.286+0000 7fbbf15d0700 1 -- 172.21.15.111:0/3657561447 learned_addr learned my addr 172.21.15.111:0/3657561447 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbf15d0700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbdc00b200 con 0x7fbbec0fad90 2024-02-19T22:13:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbf15d0700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbbdc002f40 tx=0x7fbbdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbdc01b410 con 0x7fbbec0fad90 2024-02-19T22:13:14.290 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7fbbdc01ba00 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec0ff660 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbdc01a900 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.287+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec0ffaf0 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.288+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbbdc00f070 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.288+0000 7fbbd9ffb700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.288+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fbbdc042460 con 0x7fbbec0fad90 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.288+0000 7fbbf0dcf700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 msgr2=0x7fbbd4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.288+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-02-19T22:13:14.291 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.289+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbec059100 con 0x7fbbd402ff70 2024-02-19T22:13:14.490 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.489+0000 7fbbf0dcf700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 msgr2=0x7fbbd4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:14.490 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.489+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-02-19T22:13:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.890+0000 7fbbf0dcf700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 msgr2=0x7fbbd4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:14.891 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:14.890+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-02-19T22:13:15.692 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:15.691+0000 7fbbf0dcf700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 msgr2=0x7fbbd4032430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.111:6800/959358532 2024-02-19T22:13:15.692 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:15.691+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-02-19T22:13:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:17.155+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fbbdc025980 con 0x7fbbec0fad90 2024-02-19T22:13:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:17.155+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 msgr2=0x7fbbd4032430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T22:13:17.157 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:17.155+0000 7fbbd9ffb700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:17.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:16.591278+0000 mon.smithi111 (mon.0) 88 : cluster [INF] Active manager daemon smithi111.ftjzov restarted 2024-02-19T22:13:17.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:16.591700+0000 mon.smithi111 (mon.0) 89 : cluster [INF] Activating manager daemon smithi111.ftjzov 2024-02-19T22:13:17.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:16.594051+0000 mon.smithi111 (mon.0) 90 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-02-19T22:13:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.159+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fbbdc025460 con 0x7fbbec0fad90 2024-02-19T22:13:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.159+0000 7fbbd9ffb700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 0x7fbbd4035e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.159+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbec059100 con 0x7fbbd4033a50 2024-02-19T22:13:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.159+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 0x7fbbd4035e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:18.161 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.160+0000 7fbbf0dcf700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 0x7fbbd4035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fbbe80036d0 tx=0x7fbbe8006ca0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:18.162 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.161+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fbbec059100 con 0x7fbbd4033a50 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.166+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbbec045ad0 con 0x7fbbd4033a50 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbd9ffb700 1 -- 172.21.15.111:0/3657561447 <== mgr.14162 v2:172.21.15.111:6800/1850216637 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fbbec045ad0 con 0x7fbbd4033a50 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "initialized": true 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 msgr2=0x7fbbd4035e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 0x7fbbd4035e40 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fbbe80036d0 tx=0x7fbbe8006ca0).stop 2024-02-19T22:13:18.169 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 msgr2=0x7fbbec101040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbbdc002f40 tx=0x7fbbdc006ee0).stop 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 shutdown_connections 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbd4033a50 0x7fbbd4035e40 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbec0fad90 0x7fbbec101040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 --2- 172.21.15.111:0/3657561447 >> [v2:172.21.15.111:6800/959358532,v1:172.21.15.111:6801/959358532] conn(0x7fbbd402ff70 0x7fbbd4032430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 >> 172.21.15.111:0/3657561447 conn(0x7fbbec0f6530 msgr2=0x7fbbec070050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 shutdown_connections 2024-02-19T22:13:18.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:18.167+0000 7fbbf3834700 1 -- 172.21.15.111:0/3657561447 wait complete. 2024-02-19T22:13:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:17.156589+0000 mon.smithi111 (mon.0) 91 : cluster [DBG] mgrmap e14: smithi111.ftjzov(active, starting, since 0.564969s) 2024-02-19T22:13:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.157694+0000 mon.smithi111 (mon.0) 92 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-02-19T22:13:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.158255+0000 mon.smithi111 (mon.0) 93 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mgr metadata", "who": "smithi111.ftjzov", "id": "smithi111.ftjzov"}]: dispatch 2024-02-19T22:13:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.159147+0000 mon.smithi111 (mon.0) 94 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.159307+0000 mon.smithi111 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.159458+0000 mon.smithi111 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:17.172827+0000 mon.smithi111 (mon.0) 97 : cluster [INF] Manager daemon smithi111.ftjzov is now available 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.270129+0000 mon.smithi111 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.271544+0000 mon.smithi111 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.293528+0000 mon.smithi111 (mon.0) 100 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:13:18.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:17.297016+0000 mon.smithi111 (mon.0) 101 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:mgr epoch 13 is available 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:Generating a dashboard self-signed certificate... 2024-02-19T22:13:18.789 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-02-19T22:13:19.401 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f648a577700 1 Processor -- start 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f648a577700 1 -- start start 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f648a577700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f648a577700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64840feff0 con 0x7f64840704a0 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f6483fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f6483fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39812/0 (socket says 172.21.15.111:39812) 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.400+0000 7f6483fff700 1 -- 172.21.15.111:0/2214775470 learned_addr learned my addr 172.21.15.111:0/2214775470 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f6483fff700 1 -- 172.21.15.111:0/2214775470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64840ff130 con 0x7f64840704a0 2024-02-19T22:13:19.404 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f6483fff700 1 --2- 172.21.15.111:0/2214775470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f647400b5f0 tx=0x7f6474006ee0).ready entity=mon.0 client_cookie=ba7eb59ebf93a973 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f6482ffd700 1 -- 172.21.15.111:0/2214775470 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647400d600 con 0x7f64840704a0 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f6482ffd700 1 -- 172.21.15.111:0/2214775470 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f647400dbf0 con 0x7f64840704a0 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f6482ffd700 1 -- 172.21.15.111:0/2214775470 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6474011cb0 con 0x7f64840704a0 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f648a577700 1 -- 172.21.15.111:0/2214775470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 msgr2=0x7f64840feb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f648a577700 1 --2- 172.21.15.111:0/2214775470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f647400b5f0 tx=0x7f6474006ee0).stop 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f648a577700 1 -- 172.21.15.111:0/2214775470 shutdown_connections 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.401+0000 7f648a577700 1 --2- 172.21.15.111:0/2214775470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840feb00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 -- 172.21.15.111:0/2214775470 >> 172.21.15.111:0/2214775470 conn(0x7f64840f65c0 msgr2=0x7f64840f89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 -- 172.21.15.111:0/2214775470 shutdown_connections 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 -- 172.21.15.111:0/2214775470 wait complete. 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 Processor -- start 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 -- start start 2024-02-19T22:13:19.405 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.402+0000 7f648a577700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64840faa00 con 0x7f64840704a0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6483fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6483fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:39814/0 (socket says 172.21.15.111:39814) 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6483fff700 1 -- 172.21.15.111:0/568924938 learned_addr learned my addr 172.21.15.111:0/568924938 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6483fff700 1 -- 172.21.15.111:0/568924938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647400b200 con 0x7f64840704a0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6483fff700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f647400dfc0 tx=0x7f6474006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647401b410 con 0x7f64840704a0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f647401ba00 con 0x7f64840704a0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64840faba0 con 0x7f64840704a0 2024-02-19T22:13:19.406 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f647401a990 con 0x7f64840704a0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.403+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64840fb030 con 0x7f64840704a0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.404+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f647400f070 con 0x7f64840704a0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.404+0000 7f6480ff9700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 0x7f6464032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.404+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6474041570 con 0x7f64840704a0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.404+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6484045ad0 con 0x7f64840704a0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.404+0000 7f64837fe700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 0x7f6464032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:19.407 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.405+0000 7f64837fe700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 0x7f6464032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f646c0096a0 tx=0x7f646c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:19.409 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.409+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6474016070 con 0x7f64840704a0 2024-02-19T22:13:19.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:18.161080+0000 mgr.smithi111.ftjzov (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T22:13:19.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:18.161420+0000 mon.smithi111 (mon.0) 102 : cluster [DBG] mgrmap e15: smithi111.ftjzov(active, since 1.5698s) 2024-02-19T22:13:19.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:18.167457+0000 mgr.smithi111.ftjzov (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T22:13:19.563 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.562+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f64840fb270 con 0x7f646402ff70 2024-02-19T22:13:19.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.850+0000 7f6480ff9700 1 -- 172.21.15.111:0/568924938 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f64840fb270 con 0x7f646402ff70 2024-02-19T22:13:19.851 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Self-signed certificate created 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.851+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 msgr2=0x7f6464032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.851+0000 7f648a577700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 0x7f6464032430 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f646c0096a0 tx=0x7f646c006b40).stop 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.851+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 msgr2=0x7f64840fc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.851+0000 7f648a577700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f647400dfc0 tx=0x7f6474006ee0).stop 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 shutdown_connections 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f646402ff70 0x7f6464032430 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.854 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 --2- 172.21.15.111:0/568924938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f64840704a0 0x7f64840fc580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:19.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 >> 172.21.15.111:0/568924938 conn(0x7f64840f65c0 msgr2=0x7f64840f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:19.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 shutdown_connections 2024-02-19T22:13:19.855 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:19.852+0000 7f648a577700 1 -- 172.21.15.111:0/568924938 wait complete. 2024-02-19T22:13:20.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:19.162016+0000 mon.smithi111 (mon.0) 103 : cluster [DBG] mgrmap e16: smithi111.ftjzov(active, since 2s) 2024-02-19T22:13:20.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:19.563719+0000 mgr.smithi111.ftjzov (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:20.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:19.848446+0000 mon.smithi111 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:20.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:19.850295+0000 mon.smithi111 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:20.478 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial admin user... 2024-02-19T22:13:20.479 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptd8ws58u:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.030+0000 7f3e5c87d700 1 Processor -- start 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.030+0000 7f3e5c87d700 1 -- start start 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.030+0000 7f3e5c87d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.030+0000 7f3e5c87d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580fef80 con 0x7f3e580703f0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40140/0 (socket says 172.21.15.111:40140) 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5659c700 1 -- 172.21.15.111:0/3942861467 learned_addr learned my addr 172.21.15.111:0/3942861467 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5659c700 1 -- 172.21.15.111:0/3942861467 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580ff0c0 con 0x7f3e580703f0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5659c700 1 --2- 172.21.15.111:0/3942861467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3e4c00b5f0 tx=0x7f3e4c006ee0).ready entity=mon.0 client_cookie=8a7b8b7740168efd server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5559a700 1 -- 172.21.15.111:0/3942861467 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4c00d600 con 0x7f3e580703f0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5559a700 1 -- 172.21.15.111:0/3942861467 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3e4c00dbf0 con 0x7f3e580703f0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.031+0000 7f3e5559a700 1 -- 172.21.15.111:0/3942861467 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4c011ce0 con 0x7f3e580703f0 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- 172.21.15.111:0/3942861467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 msgr2=0x7f3e580fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.034 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/3942861467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3e4c00b5f0 tx=0x7f3e4c006ee0).stop 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- 172.21.15.111:0/3942861467 shutdown_connections 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/3942861467 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e580fea90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- 172.21.15.111:0/3942861467 >> 172.21.15.111:0/3942861467 conn(0x7f3e580f6530 msgr2=0x7f3e580f8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- 172.21.15.111:0/3942861467 shutdown_connections 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- 172.21.15.111:0/3942861467 wait complete. 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 Processor -- start 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.032+0000 7f3e5c87d700 1 -- start start 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5c87d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5c87d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5810b030 con 0x7f3e580703f0 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40144/0 (socket says 172.21.15.111:40144) 2024-02-19T22:13:21.035 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5659c700 1 -- 172.21.15.111:0/210611868 learned_addr learned my addr 172.21.15.111:0/210611868 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:21.036 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5659c700 1 -- 172.21.15.111:0/210611868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c00b200 con 0x7f3e580703f0 2024-02-19T22:13:21.036 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5659c700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3e4c011ed0 tx=0x7f3e4c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4c01b570 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5810b230 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f3e4c01bb60 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e4c01aad0 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.033+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5810b6c0 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.034+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3e4c00f070 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.035+0000 7f3e46ffd700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 0x7f3e3c032430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.035+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e4c041680 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.035+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e58045ad0 con 0x7f3e580703f0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.035+0000 7f3e55d9b700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 0x7f3e3c032430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:21.037 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.035+0000 7f3e55d9b700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 0x7f3e3c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3e480096a0 tx=0x7f3e48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:21.039 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.039+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e4c016070 con 0x7f3e580703f0 2024-02-19T22:13:21.203 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.202+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3e58104ca0 con 0x7f3e3c02ff70 2024-02-19T22:13:21.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.418+0000 7f3e46ffd700 1 -- 172.21.15.111:0/210611868 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3e58104ca0 con 0x7f3e3c02ff70 2024-02-19T22:13:21.419 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$mD8ItybGOwPWQBD4.9uHEu8dcp.eD8h5BbLMk.Yex2Uz72ssvGZQ6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708380801, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 msgr2=0x7f3e3c032430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 0x7f3e3c032430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3e480096a0 tx=0x7f3e48006b40).stop 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 msgr2=0x7f3e5810ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3e4c011ed0 tx=0x7f3e4c006ee0).stop 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 shutdown_connections 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3e3c02ff70 0x7f3e3c032430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 --2- 172.21.15.111:0/210611868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3e580703f0 0x7f3e5810ab40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 >> 172.21.15.111:0/210611868 conn(0x7f3e580f6530 msgr2=0x7f3e580f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 shutdown_connections 2024-02-19T22:13:21.422 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:21.420+0000 7f3e5c87d700 1 -- 172.21.15.111:0/210611868 wait complete. 2024-02-19T22:13:22.035 INFO:teuthology.orchestra.run.smithi111.stderr:Fetching dashboard port number... 2024-02-19T22:13:22.035 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmphy7nb8hc:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp7tx050aq:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f1388da4700 1 Processor -- start 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f1388da4700 1 -- start start 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f1388da4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f1388da4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13840fc7a0 con 0x7f13840fbe70 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f138259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f138259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40156/0 (socket says 172.21.15.111:40156) 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.626+0000 7f138259c700 1 -- 172.21.15.111:0/1551710636 learned_addr learned my addr 172.21.15.111:0/1551710636 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f138259c700 1 -- 172.21.15.111:0/1551710636 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13840fc8e0 con 0x7f13840fbe70 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f138259c700 1 --2- 172.21.15.111:0/1551710636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f137800b5f0 tx=0x7f1378006ee0).ready entity=mon.0 client_cookie=8dfa82f3bb6ed210 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f138159a700 1 -- 172.21.15.111:0/1551710636 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f137800d600 con 0x7f13840fbe70 2024-02-19T22:13:22.649 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f138159a700 1 -- 172.21.15.111:0/1551710636 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f137800dbf0 con 0x7f13840fbe70 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f138159a700 1 -- 172.21.15.111:0/1551710636 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1378011cb0 con 0x7f13840fbe70 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f1388da4700 1 -- 172.21.15.111:0/1551710636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 msgr2=0x7f13840fc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.627+0000 7f1388da4700 1 --2- 172.21.15.111:0/1551710636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f137800b5f0 tx=0x7f1378006ee0).stop 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- 172.21.15.111:0/1551710636 shutdown_connections 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 --2- 172.21.15.111:0/1551710636 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f13840fc2b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- 172.21.15.111:0/1551710636 >> 172.21.15.111:0/1551710636 conn(0x7f13840f7660 msgr2=0x7f13840f9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- 172.21.15.111:0/1551710636 shutdown_connections 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- 172.21.15.111:0/1551710636 wait complete. 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 Processor -- start 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- start start 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f1388da4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138410b020 con 0x7f13840fbe70 2024-02-19T22:13:22.650 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f138259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f138259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40168/0 (socket says 172.21.15.111:40168) 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.628+0000 7f138259c700 1 -- 172.21.15.111:0/1721044855 learned_addr learned my addr 172.21.15.111:0/1721044855 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f138259c700 1 -- 172.21.15.111:0/1721044855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137800b200 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f138259c700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1378006800 tx=0x7f1378006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f137801b570 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1045+0+0 (secure 0 0 0) 0x7f137801bb60 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f137801a9f0 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138410b220 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.629+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138410b6b0 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1384045ad0 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f137800f070 con 0x7f13840fbe70 2024-02-19T22:13:22.651 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1372ffd700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 0x7f1368032480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13780415d0 con 0x7f13840fbe70 2024-02-19T22:13:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1381d9b700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 0x7f1368032480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.630+0000 7f1381d9b700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 0x7f1368032480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:22.652 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.634+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1378022310 con 0x7f13840fbe70 2024-02-19T22:13:22.708 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:21.203294+0000 mgr.smithi111.ftjzov (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:22.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:21.418855+0000 mon.smithi111 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:22.786 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.785+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f13840fdbe0 con 0x7f13840fbe70 2024-02-19T22:13:22.786 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.785+0000 7f1372ffd700 1 -- 172.21.15.111:0/1721044855 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v9) v1 ==== 112+0+5 (secure 0 0 0) 0x7f1378016070 con 0x7f13840fbe70 2024-02-19T22:13:22.786 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 8443 2024-02-19T22:13:22.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 msgr2=0x7f1368032480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 0x7f1368032480 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).stop 2024-02-19T22:13:22.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 msgr2=0x7f138410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:22.788 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f1378006800 tx=0x7f1378006ee0).stop 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 shutdown_connections 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f136802ffc0 0x7f1368032480 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 --2- 172.21.15.111:0/1721044855 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13840fbe70 0x7f138410ab30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 >> 172.21.15.111:0/1721044855 conn(0x7f13840f7660 msgr2=0x7f13840f8ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 shutdown_connections 2024-02-19T22:13:22.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-02-19T22:13:22.787+0000 7f1388da4700 1 -- 172.21.15.111:0/1721044855 wait complete. 2024-02-19T22:13:23.359 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-enabled firewalld.service 2024-02-19T22:13:23.368 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-02-19T22:13:23.369 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: systemctl is-active firewalld.service 2024-02-19T22:13:23.376 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-02-19T22:13:23.376 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-02-19T22:13:23.376 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr:Ceph Dashboard is now available at: 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: URL: https://smithi111.front.sepia.ceph.com:8443/ 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: User: admin 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: Password: kc20p6dlkf 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr:You can access the Ceph CLI with: 2024-02-19T22:13:23.380 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:Please consider enabling telemetry to help improve Ceph: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: ceph telemetry on 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:For more information see: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:Bootstrap complete. 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:Releasing lock 140198187514344 on /run/cephadm/dbbe4146-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:13:23.381 INFO:teuthology.orchestra.run.smithi111.stderr:Lock 140198187514344 released on /run/cephadm/dbbe4146-cf73-11ee-95bb-87774f69a715.lock 2024-02-19T22:13:23.413 INFO:tasks.cephadm:Fetching config... 2024-02-19T22:13:23.414 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:13:23.414 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T22:13:23.476 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T22:13:23.476 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:13:23.476 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T22:13:23.530 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T22:13:23.530 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:13:23.530 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/mon.smithi111/keyring of=/dev/stdout 2024-02-19T22:13:23.596 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T22:13:23.597 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-02-19T22:13:23.597 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T22:13:23.654 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T22:13:23.654 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQJ8pRcYDMY37GE3N9RXLEW/1HWKkelnfNzrVYuPeLt1Y3SMcUgx0ZTrK+MYsr7Jfik1mCB1wLj8oqM38HsZ1Djg0CHDqDwOxDG+MRvLEvlSV6hYVuGjeWZdkuJBNh3LNiSI4/bpPTfZ0k6odqQZ5xjQd47+vqTgMqm7a/4MYp5n0xG5EWhi9/dvfCl1dmb4F26anEeF1Zv7aYCPFFov/irslg/8NLJR4TPk+QvmsSHX077q5fxZ31KZYRbpctveoYSs0Ay3LZHVLOnPt0pgz1giHXyo6/kqu1OhGqwXNa2bRgm/bh37BFVoTe+DPF110eD2wFssfC/av5j6p95xNmy/JBTA9VGKzMezU03KgAF3C+lmPXAbGsbHEn9mva9uatsu+I0yzlK3Ei4oUlb0lNFQuHdFt1y5jxJRzKpNfFUadmzR4lI8QKzPa6rF+DNrcX8Q8TSZ8caXdZ/bNFN4/8oFaKMsxxc2IqA6xOX4kvnrZByrkEX61VMTg1pvyi/IM= ceph-dbbe4146-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:13:23.708 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:22.785708+0000 mon.smithi111 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.111:0/1721044855' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T22:13:23.730 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQJ8pRcYDMY37GE3N9RXLEW/1HWKkelnfNzrVYuPeLt1Y3SMcUgx0ZTrK+MYsr7Jfik1mCB1wLj8oqM38HsZ1Djg0CHDqDwOxDG+MRvLEvlSV6hYVuGjeWZdkuJBNh3LNiSI4/bpPTfZ0k6odqQZ5xjQd47+vqTgMqm7a/4MYp5n0xG5EWhi9/dvfCl1dmb4F26anEeF1Zv7aYCPFFov/irslg/8NLJR4TPk+QvmsSHX077q5fxZ31KZYRbpctveoYSs0Ay3LZHVLOnPt0pgz1giHXyo6/kqu1OhGqwXNa2bRgm/bh37BFVoTe+DPF110eD2wFssfC/av5j6p95xNmy/JBTA9VGKzMezU03KgAF3C+lmPXAbGsbHEn9mva9uatsu+I0yzlK3Ei4oUlb0lNFQuHdFt1y5jxJRzKpNfFUadmzR4lI8QKzPa6rF+DNrcX8Q8TSZ8caXdZ/bNFN4/8oFaKMsxxc2IqA6xOX4kvnrZByrkEX61VMTg1pvyi/IM= ceph-dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:13:23.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQJ8pRcYDMY37GE3N9RXLEW/1HWKkelnfNzrVYuPeLt1Y3SMcUgx0ZTrK+MYsr7Jfik1mCB1wLj8oqM38HsZ1Djg0CHDqDwOxDG+MRvLEvlSV6hYVuGjeWZdkuJBNh3LNiSI4/bpPTfZ0k6odqQZ5xjQd47+vqTgMqm7a/4MYp5n0xG5EWhi9/dvfCl1dmb4F26anEeF1Zv7aYCPFFov/irslg/8NLJR4TPk+QvmsSHX077q5fxZ31KZYRbpctveoYSs0Ay3LZHVLOnPt0pgz1giHXyo6/kqu1OhGqwXNa2bRgm/bh37BFVoTe+DPF110eD2wFssfC/av5j6p95xNmy/JBTA9VGKzMezU03KgAF3C+lmPXAbGsbHEn9mva9uatsu+I0yzlK3Ei4oUlb0lNFQuHdFt1y5jxJRzKpNfFUadmzR4lI8QKzPa6rF+DNrcX8Q8TSZ8caXdZ/bNFN4/8oFaKMsxxc2IqA6xOX4kvnrZByrkEX61VMTg1pvyi/IM= ceph-dbbe4146-cf73-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T22:13:23.783 INFO:teuthology.orchestra.run.smithi203.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQJ8pRcYDMY37GE3N9RXLEW/1HWKkelnfNzrVYuPeLt1Y3SMcUgx0ZTrK+MYsr7Jfik1mCB1wLj8oqM38HsZ1Djg0CHDqDwOxDG+MRvLEvlSV6hYVuGjeWZdkuJBNh3LNiSI4/bpPTfZ0k6odqQZ5xjQd47+vqTgMqm7a/4MYp5n0xG5EWhi9/dvfCl1dmb4F26anEeF1Zv7aYCPFFov/irslg/8NLJR4TPk+QvmsSHX077q5fxZ31KZYRbpctveoYSs0Ay3LZHVLOnPt0pgz1giHXyo6/kqu1OhGqwXNa2bRgm/bh37BFVoTe+DPF110eD2wFssfC/av5j6p95xNmy/JBTA9VGKzMezU03KgAF3C+lmPXAbGsbHEn9mva9uatsu+I0yzlK3Ei4oUlb0lNFQuHdFt1y5jxJRzKpNfFUadmzR4lI8QKzPa6rF+DNrcX8Q8TSZ8caXdZ/bNFN4/8oFaKMsxxc2IqA6xOX4kvnrZByrkEX61VMTg1pvyi/IM= ceph-dbbe4146-cf73-11ee-95bb-87774f69a715 2024-02-19T22:13:23.795 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T22:13:24.458 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:23.421214+0000 mon.smithi111 (mon.0) 108 : cluster [DBG] mgrmap e17: smithi111.ftjzov(active, since 6s) 2024-02-19T22:13:26.717 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1206149035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 msgr2=0x7f5a6c0fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1206149035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c0fb550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f5a5c009fb0 tx=0x7f5a5c009b30).stop 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1206149035 shutdown_connections 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1206149035 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c0fb550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1206149035 >> 172.21.15.111:0/1206149035 conn(0x7f5a6c0f0b70 msgr2=0x7f5a6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1206149035 shutdown_connections 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.715+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1206149035 wait complete. 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a70cc5700 1 Processor -- start 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a70cc5700 1 -- start start 2024-02-19T22:13:26.718 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a70cc5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a70cc5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a6c108ec0 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a6a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:40196/0 (socket says 172.21.15.111:40196) 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.716+0000 7f5a6a59c700 1 -- 172.21.15.111:0/1969599326 learned_addr learned my addr 172.21.15.111:0/1969599326 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a6a59c700 1 -- 172.21.15.111:0/1969599326 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a5c005040 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a6a59c700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5a5c009830 tx=0x7f5a5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a5c009320 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5a5c013380 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a5c014420 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c1090c0 con 0x7f5a6c060220 2024-02-19T22:13:26.719 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.717+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c109550 con 0x7f5a6c060220 2024-02-19T22:13:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a5c014580 con 0x7f5a6c060220 2024-02-19T22:13:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c045ad0 con 0x7f5a6c060220 2024-02-19T22:13:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a5affd700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 0x7f5a54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:26.720 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a5c041150 con 0x7f5a6c060220 2024-02-19T22:13:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a69d9b700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 0x7f5a54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:26.721 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.718+0000 7f5a69d9b700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 0x7f5a54032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5a600096a0 tx=0x7f5a60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:26.723 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.722+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a5c01c8d0 con 0x7f5a6c060220 2024-02-19T22:13:26.890 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.890+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5a6c0fe290 con 0x7f5a6c060220 2024-02-19T22:13:26.893 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.893+0000 7f5a5affd700 1 -- 172.21.15.111:0/1969599326 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5a5c01c6a0 con 0x7f5a6c060220 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 msgr2=0x7f5a54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 0x7f5a54032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f5a600096a0 tx=0x7f5a60006b40).stop 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 msgr2=0x7f5a6c1089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f5a5c009830 tx=0x7f5a5c00b040).stop 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 shutdown_connections 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5a5402ffb0 0x7f5a54032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 --2- 172.21.15.111:0/1969599326 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5a6c060220 0x7f5a6c1089d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 >> 172.21.15.111:0/1969599326 conn(0x7f5a6c0f0b70 msgr2=0x7f5a6c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:26.896 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 shutdown_connections 2024-02-19T22:13:26.897 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:26.895+0000 7f5a70cc5700 1 -- 172.21.15.111:0/1969599326 wait complete. 2024-02-19T22:13:27.629 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:26.259856+0000 mon.smithi111 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:27.629 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:26.518926+0000 mon.smithi111 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:27.629 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:26.893338+0000 mon.smithi111 (mon.0) 111 : audit [INF] from='client.? 172.21.15.111:0/1969599326' entity='client.admin' 2024-02-19T22:13:27.650 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi203 2024-02-19T22:13:27.650 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:13:27.650 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.conf 2024-02-19T22:13:27.672 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-02-19T22:13:27.672 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T22:13:27.733 INFO:tasks.cephadm:Adding host smithi203 to orchestrator... 2024-02-19T22:13:27.733 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph orch host add smithi203 2024-02-19T22:13:29.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:29 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:28.133172+0000 mon.smithi111 (mon.0) 112 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:32.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:31.484606+0000 mon.smithi111 (mon.0) 113 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:32.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:31.740022+0000 mon.smithi111 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:32.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:31.745036+0000 mon.smithi111 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:33.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:31.750980+0000 mgr.smithi111.ftjzov (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi111 on smithi111 2024-02-19T22:13:38.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:37.160468+0000 mgr.smithi111.ftjzov (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:40.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:39.160784+0000 mgr.smithi111.ftjzov (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:41.161157+0000 mgr.smithi111.ftjzov (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:43.161510+0000 mgr.smithi111.ftjzov (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:45.161823+0000 mgr.smithi111.ftjzov (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:47.162007+0000 mgr.smithi111.ftjzov (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 -- 172.21.15.203:0/837554897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 msgr2=0x7f41680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 --2- 172.21.15.203:0/837554897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41680f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f4158009fb0 tx=0x7f4158009b30).stop 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 -- 172.21.15.203:0/837554897 shutdown_connections 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 --2- 172.21.15.203:0/837554897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41680f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 -- 172.21.15.203:0/837554897 >> 172.21.15.203:0/837554897 conn(0x7f41680f0b70 msgr2=0x7f41680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 -- 172.21.15.203:0/837554897 shutdown_connections 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.591+0000 7f41704c9700 1 -- 172.21.15.203:0/837554897 wait complete. 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f41704c9700 1 Processor -- start 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f41704c9700 1 -- start start 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f41704c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f41704c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4168105bd0 con 0x7f41680f5410 2024-02-19T22:13:49.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f416e265700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f416e265700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53562/0 (socket says 172.21.15.203:53562) 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.592+0000 7f416e265700 1 -- 172.21.15.203:0/3156012915 learned_addr learned my addr 172.21.15.203:0/3156012915 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f416e265700 1 -- 172.21.15.203:0/3156012915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158005040 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f416e265700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4158005010 tx=0x7f415800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4158009320 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4168105dd0 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4158013380 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4158014e50 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.593+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4168106260 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4158014420 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f415effd700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 0x7f41540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4168045ad0 con 0x7f41680f5410 2024-02-19T22:13:49.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4158040e30 con 0x7f41680f5410 2024-02-19T22:13:49.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f416da64700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 0x7f41540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:49.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.594+0000 7f416da64700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 0x7f41540323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f41640096a0 tx=0x7f4164006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:49.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.599+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f415801db70 con 0x7f41680f5410 2024-02-19T22:13:49.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:49.771+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7f41680f1d20 con 0x7f415402ff10 2024-02-19T22:13:50.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.382+0000 7f415effd700 1 -- 172.21.15.203:0/3156012915 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f41680f1d20 con 0x7f415402ff10 2024-02-19T22:13:50.384 INFO:teuthology.orchestra.run.smithi203.stdout:Added host 'smithi203' 2024-02-19T22:13:50.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 msgr2=0x7f41540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 0x7f41540323d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f41640096a0 tx=0x7f4164006b40).stop 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 msgr2=0x7f41681056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4158005010 tx=0x7f415800b040).stop 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 shutdown_connections 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f415402ff10 0x7f41540323d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 --2- 172.21.15.203:0/3156012915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f41680f5410 0x7f41681056e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.384+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 >> 172.21.15.203:0/3156012915 conn(0x7f41680f0b70 msgr2=0x7f41680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.385+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 shutdown_connections 2024-02-19T22:13:50.387 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:50.385+0000 7f41704c9700 1 -- 172.21.15.203:0/3156012915 wait complete. 2024-02-19T22:13:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:49.162319+0000 mgr.smithi111.ftjzov (mgr.14162) 12 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:51.115 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T22:13:51.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:49.772930+0000 mgr.smithi111.ftjzov (mgr.14162) 13 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:13:51.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:50.382849+0000 mon.smithi111 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:13:51.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:50.383257+0000 mgr.smithi111.ftjzov (mgr.14162) 14 : cephadm [INF] Added host smithi203 2024-02-19T22:13:52.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:51.162563+0000 mgr.smithi111.ftjzov (mgr.14162) 15 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- 172.21.15.203:0/3713126839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 msgr2=0x7f50d80f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3713126839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d80f8fc0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f50cc009fb0 tx=0x7f50cc009b30).stop 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- 172.21.15.203:0/3713126839 shutdown_connections 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3713126839 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d80f8fc0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- 172.21.15.203:0/3713126839 >> 172.21.15.203:0/3713126839 conn(0x7f50d80f09a0 msgr2=0x7f50d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- 172.21.15.203:0/3713126839 shutdown_connections 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- 172.21.15.203:0/3713126839 wait complete. 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 Processor -- start 2024-02-19T22:13:52.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.931+0000 7f50ddc95700 1 -- start start 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50ddc95700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50ddc95700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d8106af0 con 0x7f50d8060140 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50dcc93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50dcc93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53572/0 (socket says 172.21.15.203:53572) 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50dcc93700 1 -- 172.21.15.203:0/3100827058 learned_addr learned my addr 172.21.15.203:0/3100827058 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50dcc93700 1 -- 172.21.15.203:0/3100827058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc005040 con 0x7f50d8060140 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50dcc93700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f50cc005010 tx=0x7f50cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50cc009320 con 0x7f50d8060140 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d8106cf0 con 0x7f50d8060140 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d8107180 con 0x7f50d8060140 2024-02-19T22:13:52.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50cc013380 con 0x7f50d8060140 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.932+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50cc014e50 con 0x7f50d8060140 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.933+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50cc0095f0 con 0x7f50d8060140 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.933+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d8045ad0 con 0x7f50d8060140 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.933+0000 7f50d57fa700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 0x7f50c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.933+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50cc040f70 con 0x7f50d8060140 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.934+0000 7f50d7fff700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 0x7f50c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.934+0000 7f50d7fff700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 0x7f50c80323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f50c40096a0 tx=0x7f50c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:52.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:52.938+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50cc0268c0 con 0x7f50d8060140 2024-02-19T22:13:53.111 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.110+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50d80f1b70 con 0x7f50c802ff10 2024-02-19T22:13:53.113 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.112+0000 7f50d57fa700 1 -- 172.21.15.203:0/3100827058 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f50d80f1b70 con 0x7f50c802ff10 2024-02-19T22:13:53.113 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:13:53.113 INFO:teuthology.orchestra.run.smithi203.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "smithi203", "hostname": "smithi203", "labels": [], "status": ""}] 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 msgr2=0x7f50c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 0x7f50c80323d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f50c40096a0 tx=0x7f50c4006b40).stop 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 msgr2=0x7f50d8106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f50cc005010 tx=0x7f50cc00b040).stop 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 shutdown_connections 2024-02-19T22:13:53.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f50c802ff10 0x7f50c80323d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:53.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 --2- 172.21.15.203:0/3100827058 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50d8060140 0x7f50d8106600 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:53.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 >> 172.21.15.203:0/3100827058 conn(0x7f50d80f09a0 msgr2=0x7f50d80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:53.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 shutdown_connections 2024-02-19T22:13:53.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:53.113+0000 7f50ddc95700 1 -- 172.21.15.203:0/3100827058 wait complete. 2024-02-19T22:13:53.713 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T22:13:53.713 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T22:13:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:53.112343+0000 mgr.smithi111.ftjzov (mgr.14162) 16 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T22:13:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:53.162815+0000 mgr.smithi111.ftjzov (mgr.14162) 17 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:55.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 -- 172.21.15.111:0/3787205956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 msgr2=0x7f35740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:55.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 --2- 172.21.15.111:0/3787205956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35740f55c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3568009fb0 tx=0x7f3568009b30).stop 2024-02-19T22:13:55.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 -- 172.21.15.111:0/3787205956 shutdown_connections 2024-02-19T22:13:55.585 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 --2- 172.21.15.111:0/3787205956 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35740f55c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 -- 172.21.15.111:0/3787205956 >> 172.21.15.111:0/3787205956 conn(0x7f35740f09a0 msgr2=0x7f35740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 -- 172.21.15.111:0/3787205956 shutdown_connections 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.581+0000 7f357920b700 1 -- 172.21.15.111:0/3787205956 wait complete. 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f357920b700 1 Processor -- start 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f357920b700 1 -- start start 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f357920b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f357920b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3574106ac0 con 0x7f35740f51a0 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3573fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3573fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.111:56466/0 (socket says 172.21.15.111:56466) 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3573fff700 1 -- 172.21.15.111:0/147525479 learned_addr learned my addr 172.21.15.111:0/147525479 (peer_addr_for_me v2:172.21.15.111:0/0) 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3573fff700 1 -- 172.21.15.111:0/147525479 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568005040 con 0x7f35740f51a0 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3573fff700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3568009830 tx=0x7f356800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.586 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.582+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3568009320 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3568013380 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3568014420 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3574106cc0 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3574107150 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3568014580 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f3570ff9700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 0x7f355c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.583+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3568040ec0 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.584+0000 7f35737fe700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 0x7f355c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.584+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3574059100 con 0x7f35740f51a0 2024-02-19T22:13:55.587 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.584+0000 7f35737fe700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 0x7f355c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f35640096a0 tx=0x7f3564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:55.589 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.588+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35680273f0 con 0x7f35740f51a0 2024-02-19T22:13:55.745 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:55.743+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3574045ad0 con 0x7f35740f51a0 2024-02-19T22:13:56.390 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.388+0000 7f3570ff9700 1 -- 172.21.15.111:0/147525479 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f356801c890 con 0x7f35740f51a0 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 msgr2=0x7f355c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 0x7f355c032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f35640096a0 tx=0x7f3564006b40).stop 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 msgr2=0x7f35741065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f3568009830 tx=0x7f356800b040).stop 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 shutdown_connections 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f355c02ff60 0x7f355c032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 --2- 172.21.15.111:0/147525479 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35740f51a0 0x7f35741065d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 >> 172.21.15.111:0/147525479 conn(0x7f35740f09a0 msgr2=0x7f357405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 shutdown_connections 2024-02-19T22:13:56.393 INFO:teuthology.orchestra.run.smithi111.stderr:2024-02-19T22:13:56.391+0000 7f357920b700 1 -- 172.21.15.111:0/147525479 wait complete. 2024-02-19T22:13:56.394 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2024-02-19T22:13:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:55.163098+0000 mgr.smithi111.ftjzov (mgr.14162) 18 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:55.744857+0000 mon.smithi111 (mon.0) 117 : audit [INF] from='client.? 172.21.15.111:0/147525479' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T22:13:57.057 INFO:tasks.cephadm:Adding mon.smithi111 on smithi111 2024-02-19T22:13:57.057 INFO:tasks.cephadm:Adding mon.smithi203 on smithi203 2024-02-19T22:13:57.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi111:172.21.15.111=smithi111;smithi203:172.21.15.203=smithi203' 2024-02-19T22:13:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:56.390184+0000 mon.smithi111 (mon.0) 118 : audit [INF] from='client.? 172.21.15.111:0/147525479' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T22:13:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:56.390234+0000 mon.smithi111 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-02-19T22:13:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:13:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:57.163359+0000 mgr.smithi111.ftjzov (mgr.14162) 19 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.941+0000 7f0fb359e700 1 -- 172.21.15.203:0/811094253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 msgr2=0x7f0fb40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.941+0000 7f0fb359e700 1 --2- 172.21.15.203:0/811094253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb40f55c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f0f9c009fb0 tx=0x7f0f9c009b30).stop 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- 172.21.15.203:0/811094253 shutdown_connections 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 --2- 172.21.15.203:0/811094253 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb40f55c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- 172.21.15.203:0/811094253 >> 172.21.15.203:0/811094253 conn(0x7f0fb40f09a0 msgr2=0x7f0fb40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:58.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- 172.21.15.203:0/811094253 shutdown_connections 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- 172.21.15.203:0/811094253 wait complete. 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 Processor -- start 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- start start 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.942+0000 7f0fb359e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb41059c0 con 0x7f0fb40f51a0 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:36090/0 (socket says 172.21.15.203:36090) 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb259c700 1 -- 172.21.15.203:0/2366073499 learned_addr learned my addr 172.21.15.203:0/2366073499 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb259c700 1 -- 172.21.15.203:0/2366073499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c005040 con 0x7f0fb40f51a0 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb259c700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0f9c0089f0 tx=0x7f0f9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f9c009320 con 0x7f0fb40f51a0 2024-02-19T22:13:58.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb4105bc0 con 0x7f0fb40f51a0 2024-02-19T22:13:58.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0f9c013380 con 0x7f0fb40f51a0 2024-02-19T22:13:58.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f9c014e20 con 0x7f0fb40f51a0 2024-02-19T22:13:58.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.943+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb4106050 con 0x7f0fb40f51a0 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.944+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb4045ad0 con 0x7f0fb40f51a0 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.944+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f9c0095f0 con 0x7f0fb40f51a0 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.944+0000 7f0faaffd700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 0x7f0fa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.944+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f9c041510 con 0x7f0fb40f51a0 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.945+0000 7f0fb1d9b700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 0x7f0fa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:13:58.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.945+0000 7f0fb1d9b700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 0x7f0fa00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0fa40096a0 tx=0x7f0fa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:13:58.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:58.949+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f9c014420 con 0x7f0fb40f51a0 2024-02-19T22:13:59.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.124+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 --> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi203:172.21.15.203=smithi203", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fb40f1b70 con 0x7f0fa002ff10 2024-02-19T22:13:59.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.130+0000 7f0faaffd700 1 -- 172.21.15.203:0/2366073499 <== mgr.14162 v2:172.21.15.111:6800/1850216637 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0fb40f1b70 con 0x7f0fa002ff10 2024-02-19T22:13:59.131 INFO:teuthology.orchestra.run.smithi203.stdout:Scheduled mon update... 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 msgr2=0x7f0fa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 0x7f0fa00323d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0fa40096a0 tx=0x7f0fa4006b40).stop 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 msgr2=0x7f0fb41054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f0f9c0089f0 tx=0x7f0f9c00b040).stop 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 shutdown_connections 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0fa002ff10 0x7f0fa00323d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 --2- 172.21.15.203:0/2366073499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0fb40f51a0 0x7f0fb41054d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 >> 172.21.15.203:0/2366073499 conn(0x7f0fb40f09a0 msgr2=0x7f0fb40f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 shutdown_connections 2024-02-19T22:13:59.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:13:59.131+0000 7f0fb359e700 1 -- 172.21.15.203:0/2366073499 wait complete. 2024-02-19T22:13:59.865 DEBUG:teuthology.orchestra.run.smithi203:mon.smithi203> sudo journalctl -f -n 0 -u ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203.service 2024-02-19T22:13:59.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:13:59.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:13:59.895 INFO:journalctl@ceph.mon.smithi203.smithi203.stdout:-- Logs begin at Mon 2024-02-19 21:54:36 UTC. -- 2024-02-19T22:14:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:59.126275+0000 mgr.smithi111.ftjzov (mgr.14162) 20 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi203:172.21.15.203=smithi203", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T22:14:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:13:59.127711+0000 mgr.smithi111.ftjzov (mgr.14162) 21 : cephadm [INF] Saving service mon spec with placement smithi111:172.21.15.111=smithi111;smithi203:172.21.15.203=smithi203;count:2 2024-02-19T22:14:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:13:59.130413+0000 mon.smithi111 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:14:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:13:59.163597+0000 mgr.smithi111.ftjzov (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 -- 172.21.15.203:0/2980698792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 msgr2=0x7f06fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 --2- 172.21.15.203:0/2980698792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc0f6930 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f06f0009fb0 tx=0x7f06f0009b30).stop 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 -- 172.21.15.203:0/2980698792 shutdown_connections 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 --2- 172.21.15.203:0/2980698792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc0f6930 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 -- 172.21.15.203:0/2980698792 >> 172.21.15.203:0/2980698792 conn(0x7f06fc0f1c70 msgr2=0x7f06fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.684+0000 7f0703516700 1 -- 172.21.15.203:0/2980698792 shutdown_connections 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f0703516700 1 -- 172.21.15.203:0/2980698792 wait complete. 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f0703516700 1 Processor -- start 2024-02-19T22:14:01.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f0703516700 1 -- start start 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f0703516700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f0703516700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06fc106cd0 con 0x7f06fc0f6510 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f07012b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f07012b2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:36108/0 (socket says 172.21.15.203:36108) 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.685+0000 7f07012b2700 1 -- 172.21.15.203:0/3830783958 learned_addr learned my addr 172.21.15.203:0/3830783958 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f07012b2700 1 -- 172.21.15.203:0/3830783958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0005040 con 0x7f06fc0f6510 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f07012b2700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f06f0005010 tx=0x7f06f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f0009320 con 0x7f06fc0f6510 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc106ed0 con 0x7f06fc0f6510 2024-02-19T22:14:01.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06f0013380 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06f0014e50 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.686+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06fc107360 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.687+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06f0014420 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.687+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06fc045ad0 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.687+0000 7f06edffb700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 0x7f06e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.687+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06f0040f70 con 0x7f06fc0f6510 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.687+0000 7f0700ab1700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 0x7f06e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.688+0000 7f0700ab1700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 0x7f06e80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f06f80096a0 tx=0x7f06f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:01.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.692+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f06f00095f0 con 0x7f06fc0f6510 2024-02-19T22:14:01.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.901+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06fc1081b0 con 0x7f06fc0f6510 2024-02-19T22:14:01.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.902+0000 7f06edffb700 1 -- 172.21.15.203:0/3830783958 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f06f001db70 con 0x7f06fc0f6510 2024-02-19T22:14:01.904 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:01.904 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.903+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 msgr2=0x7f06e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.903+0000 7f0703516700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 0x7f06e80323d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f06f80096a0 tx=0x7f06f8006b40).stop 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.903+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 msgr2=0x7f06fc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.903+0000 7f0703516700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f06f0005010 tx=0x7f06f000b040).stop 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 shutdown_connections 2024-02-19T22:14:01.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f06e802ff10 0x7f06e80323d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:01.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 --2- 172.21.15.203:0/3830783958 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f06fc0f6510 0x7f06fc1067e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:01.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 >> 172.21.15.203:0/3830783958 conn(0x7f06fc0f1c70 msgr2=0x7f06fc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:01.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 shutdown_connections 2024-02-19T22:14:01.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:01.904+0000 7f0703516700 1 -- 172.21.15.203:0/3830783958 wait complete. 2024-02-19T22:14:01.907 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:01.163947+0000 mgr.smithi111.ftjzov (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:01.903006+0000 mon.smithi111 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.203:0/3830783958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:03.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:03.506 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:03.164189+0000 mgr.smithi111.ftjzov (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:05.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- 172.21.15.203:0/1635717069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 msgr2=0x7f59e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 --2- 172.21.15.203:0/1635717069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e40f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f59d4009fb0 tx=0x7f59d4009b30).stop 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- 172.21.15.203:0/1635717069 shutdown_connections 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 --2- 172.21.15.203:0/1635717069 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e40f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- 172.21.15.203:0/1635717069 >> 172.21.15.203:0/1635717069 conn(0x7f59e40f0b70 msgr2=0x7f59e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- 172.21.15.203:0/1635717069 shutdown_connections 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- 172.21.15.203:0/1635717069 wait complete. 2024-02-19T22:14:05.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 Processor -- start 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.851+0000 7f59e96e8700 1 -- start start 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e96e8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e96e8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4106d00 con 0x7f59e40f5410 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e2ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54670/0 (socket says 172.21.15.203:54670) 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e2ffd700 1 -- 172.21.15.203:0/876234463 learned_addr learned my addr 172.21.15.203:0/876234463 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e2ffd700 1 -- 172.21.15.203:0/876234463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4005040 con 0x7f59e40f5410 2024-02-19T22:14:05.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e2ffd700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f59d4005010 tx=0x7f59d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d4009320 con 0x7f59e40f5410 2024-02-19T22:14:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d4013380 con 0x7f59e40f5410 2024-02-19T22:14:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e4106f00 con 0x7f59e40f5410 2024-02-19T22:14:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d4014e50 con 0x7f59e40f5410 2024-02-19T22:14:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.852+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e4107390 con 0x7f59e40f5410 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.853+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59d4014420 con 0x7f59e40f5410 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.853+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e4045ad0 con 0x7f59e40f5410 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.853+0000 7f59cb7fe700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 0x7f59cc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.854+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59d4040e60 con 0x7f59e40f5410 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.854+0000 7f59e27fc700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 0x7f59cc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.854+0000 7f59e27fc700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 0x7f59cc0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f59d80096a0 tx=0x7f59d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:05.858+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59d40095f0 con 0x7f59e40f5410 2024-02-19T22:14:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.064+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59e40fafa0 con 0x7f59e40f5410 2024-02-19T22:14:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.064+0000 7f59cb7fe700 1 -- 172.21.15.203:0/876234463 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59d401db70 con 0x7f59e40f5410 2024-02-19T22:14:06.066 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:06.066 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 msgr2=0x7f59cc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 0x7f59cc0323d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f59d80096a0 tx=0x7f59d8006b40).stop 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 msgr2=0x7f59e4106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f59d4005010 tx=0x7f59d400b040).stop 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 shutdown_connections 2024-02-19T22:14:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f59cc02ff10 0x7f59cc0323d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 --2- 172.21.15.203:0/876234463 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59e40f5410 0x7f59e4106810 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 >> 172.21.15.203:0/876234463 conn(0x7f59e40f0b70 msgr2=0x7f59e405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 shutdown_connections 2024-02-19T22:14:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:06.066+0000 7f59e96e8700 1 -- 172.21.15.203:0/876234463 wait complete. 2024-02-19T22:14:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:05.164529+0000 mgr.smithi111.ftjzov (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:06.065651+0000 mon.smithi111 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.203:0/876234463' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:07.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:07.750 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:07.164778+0000 mgr.smithi111.ftjzov (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:09.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 -- 172.21.15.203:0/842267606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 msgr2=0x7facec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:09.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 --2- 172.21.15.203:0/842267606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec0f6930 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7facd4009fb0 tx=0x7facd4009b30).stop 2024-02-19T22:14:09.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 -- 172.21.15.203:0/842267606 shutdown_connections 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 --2- 172.21.15.203:0/842267606 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec0f6930 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 -- 172.21.15.203:0/842267606 >> 172.21.15.203:0/842267606 conn(0x7facec0f1c70 msgr2=0x7facec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 -- 172.21.15.203:0/842267606 shutdown_connections 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 -- 172.21.15.203:0/842267606 wait complete. 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.725+0000 7facf23ad700 1 Processor -- start 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facf23ad700 1 -- start start 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facf23ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:09.728 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facf23ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec108ef0 con 0x7facec0f6510 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54684/0 (socket says 172.21.15.203:54684) 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facebfff700 1 -- 172.21.15.203:0/3617237185 learned_addr learned my addr 172.21.15.203:0/3617237185 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facebfff700 1 -- 172.21.15.203:0/3617237185 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd4005040 con 0x7facec0f6510 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.726+0000 7facebfff700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7facd4005010 tx=0x7facd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.727+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd4009320 con 0x7facec0f6510 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.727+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facec1090f0 con 0x7facec0f6510 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.727+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7facd4013380 con 0x7facec0f6510 2024-02-19T22:14:09.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.727+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7facd4014e50 con 0x7facec0f6510 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.727+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facec109580 con 0x7facec0f6510 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facec045ad0 con 0x7facec0f6510 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7facd40095f0 con 0x7facec0f6510 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7face8ff9700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 0x7facd80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7facd4040e70 con 0x7facec0f6510 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7faceb7fe700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 0x7facd80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:09.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.728+0000 7faceb7fe700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 0x7facd80323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7facdc0096a0 tx=0x7facdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:09.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.732+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7facd4014420 con 0x7facec0f6510 2024-02-19T22:14:09.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.939+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facec10a3d0 con 0x7facec0f6510 2024-02-19T22:14:09.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.940+0000 7face8ff9700 1 -- 172.21.15.203:0/3617237185 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7facd4010230 con 0x7facec0f6510 2024-02-19T22:14:09.942 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:09.942 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 msgr2=0x7facd80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 0x7facd80323d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7facdc0096a0 tx=0x7facdc006b40).stop 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 msgr2=0x7facec108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7facd4005010 tx=0x7facd400b040).stop 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 shutdown_connections 2024-02-19T22:14:09.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7facd802ff10 0x7facd80323d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:09.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 --2- 172.21.15.203:0/3617237185 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7facec0f6510 0x7facec108a00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:09.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 >> 172.21.15.203:0/3617237185 conn(0x7facec0f1c70 msgr2=0x7facec0f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:09.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 shutdown_connections 2024-02-19T22:14:09.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:09.941+0000 7facf23ad700 1 -- 172.21.15.203:0/3617237185 wait complete. 2024-02-19T22:14:09.945 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:09.165144+0000 mgr.smithi111.ftjzov (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:09.940780+0000 mon.smithi111 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.203:0/3617237185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:11.165474+0000 mgr.smithi111.ftjzov (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:14.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:14.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:13.165741+0000 mgr.smithi111.ftjzov (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:15.166085+0000 mgr.smithi111.ftjzov (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:17.166385+0000 mgr.smithi111.ftjzov (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:17.296037+0000 mon.smithi111 (mon.0) 124 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:14:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:17.298723+0000 mon.smithi111 (mon.0) 125 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:14:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:19.166691+0000 mgr.smithi111.ftjzov (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:22.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:21.167012+0000 mgr.smithi111.ftjzov (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.080+0000 7fd78f7a0700 1 -- 172.21.15.203:0/154649380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 msgr2=0x7fd7880f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.080+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/154649380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd7880f5d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd784009fb0 tx=0x7fd784009b30).stop 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.080+0000 7fd78f7a0700 1 -- 172.21.15.203:0/154649380 shutdown_connections 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.080+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/154649380 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd7880f5d70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.080+0000 7fd78f7a0700 1 -- 172.21.15.203:0/154649380 >> 172.21.15.203:0/154649380 conn(0x7fd7880f0260 msgr2=0x7fd7880f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 -- 172.21.15.203:0/154649380 shutdown_connections 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 -- 172.21.15.203:0/154649380 wait complete. 2024-02-19T22:14:23.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 Processor -- start 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 -- start start 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78f7a0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880f5310 con 0x7fd7880f5950 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78d53c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78d53c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:35412/0 (socket says 172.21.15.203:35412) 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.081+0000 7fd78d53c700 1 -- 172.21.15.203:0/1673518857 learned_addr learned my addr 172.21.15.203:0/1673518857 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd78d53c700 1 -- 172.21.15.203:0/1673518857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784005040 con 0x7fd7880f5950 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd78d53c700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd784005010 tx=0x7fd78400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd784009320 con 0x7fd7880f5950 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7880f5510 con 0x7fd7880f5950 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd784013380 con 0x7fd7880f5950 2024-02-19T22:14:23.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd784014e50 con 0x7fd7880f5950 2024-02-19T22:14:23.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.082+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd788109bb0 con 0x7fd7880f5950 2024-02-19T22:14:23.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.083+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7840095f0 con 0x7fd7880f5950 2024-02-19T22:14:23.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.083+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd788059100 con 0x7fd7880f5950 2024-02-19T22:14:23.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.083+0000 7fd77dffb700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 0x7fd7740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:23.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.083+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd784040e70 con 0x7fd7880f5950 2024-02-19T22:14:23.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.083+0000 7fd78cd3b700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 0x7fd7740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:23.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.084+0000 7fd78cd3b700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 0x7fd7740323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd7780096a0 tx=0x7fd778006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:23.088 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.088+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd784014420 con 0x7fd7880f5950 2024-02-19T22:14:23.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.293+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7880faf80 con 0x7fd7880f5950 2024-02-19T22:14:23.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.294+0000 7fd77dffb700 1 -- 172.21.15.203:0/1673518857 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd784010230 con 0x7fd7880f5950 2024-02-19T22:14:23.295 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:23.295 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:23.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 msgr2=0x7fd7740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:23.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 0x7fd7740323d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fd7780096a0 tx=0x7fd778006b40).stop 2024-02-19T22:14:23.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 msgr2=0x7fd788109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:23.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd784005010 tx=0x7fd78400b040).stop 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 shutdown_connections 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77402ff10 0x7fd7740323d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 --2- 172.21.15.203:0/1673518857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7880f5950 0x7fd788109500 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 >> 172.21.15.203:0/1673518857 conn(0x7fd7880f0260 msgr2=0x7fd7880f0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 shutdown_connections 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:23.295+0000 7fd78f7a0700 1 -- 172.21.15.203:0/1673518857 wait complete. 2024-02-19T22:14:23.297 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:23.167393+0000 mgr.smithi111.ftjzov (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:23.294105+0000 mon.smithi111 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.203:0/1673518857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:24.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:24.853 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:26.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.678+0000 7f097eee5700 1 -- 172.21.15.203:0/860845750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 msgr2=0x7f09780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.678+0000 7f097eee5700 1 --2- 172.21.15.203:0/860845750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09780f66c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f0970009fb0 tx=0x7f0970009b30).stop 2024-02-19T22:14:26.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.678+0000 7f097eee5700 1 -- 172.21.15.203:0/860845750 shutdown_connections 2024-02-19T22:14:26.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.678+0000 7f097eee5700 1 --2- 172.21.15.203:0/860845750 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09780f66c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.678+0000 7f097eee5700 1 -- 172.21.15.203:0/860845750 >> 172.21.15.203:0/860845750 conn(0x7f09780f1ac0 msgr2=0x7f09780f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 -- 172.21.15.203:0/860845750 shutdown_connections 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 -- 172.21.15.203:0/860845750 wait complete. 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 Processor -- start 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 -- start start 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.679+0000 7f097eee5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978106ab0 con 0x7f09780f62a0 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097dee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097dee3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59464/0 (socket says 172.21.15.203:59464) 2024-02-19T22:14:26.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097dee3700 1 -- 172.21.15.203:0/3673271897 learned_addr learned my addr 172.21.15.203:0/3673271897 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:26.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097dee3700 1 -- 172.21.15.203:0/3673271897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970005040 con 0x7f09780f62a0 2024-02-19T22:14:26.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097dee3700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f09700089f0 tx=0x7f097000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0970009320 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0970013380 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0978106cb0 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0970014e20 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.680+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0978107140 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.681+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f09700263a0 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.681+0000 7f09767fc700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 0x7f09600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.681+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0970040f90 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.681+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0978045ad0 con 0x7f09780f62a0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.682+0000 7f097d6e2700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 0x7f09600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:26.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.682+0000 7f097d6e2700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 0x7f09600323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f096c0096a0 tx=0x7f096c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:26.686 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.686+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0970012920 con 0x7f09780f62a0 2024-02-19T22:14:26.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:25.167728+0000 mgr.smithi111.ftjzov (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:26.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.892+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09780fbf20 con 0x7f09780f62a0 2024-02-19T22:14:26.893 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.892+0000 7f09767fc700 1 -- 172.21.15.203:0/3673271897 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f097002b020 con 0x7f09780f62a0 2024-02-19T22:14:26.893 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:26.894 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:26.895 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 msgr2=0x7f09600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.895 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 0x7f09600323d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f096c0096a0 tx=0x7f096c006b40).stop 2024-02-19T22:14:26.895 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 msgr2=0x7f09781065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:26.895 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f09700089f0 tx=0x7f097000b040).stop 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 shutdown_connections 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f096002ff10 0x7f09600323d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 --2- 172.21.15.203:0/3673271897 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09780f62a0 0x7f09781065c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 >> 172.21.15.203:0/3673271897 conn(0x7f09780f1ac0 msgr2=0x7f09780f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 shutdown_connections 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:26.894+0000 7f097eee5700 1 -- 172.21.15.203:0/3673271897 wait complete. 2024-02-19T22:14:26.896 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:27.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:26.892650+0000 mon.smithi111 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.203:0/3673271897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:28.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:28.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:27.167997+0000 mgr.smithi111.ftjzov (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:30.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 -- 172.21.15.203:0/176352828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 msgr2=0x7fc8480f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 --2- 172.21.15.203:0/176352828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc8480f66c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fc83c009fb0 tx=0x7fc83c009b30).stop 2024-02-19T22:14:30.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 -- 172.21.15.203:0/176352828 shutdown_connections 2024-02-19T22:14:30.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 --2- 172.21.15.203:0/176352828 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc8480f66c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 -- 172.21.15.203:0/176352828 >> 172.21.15.203:0/176352828 conn(0x7fc8480f1aa0 msgr2=0x7fc8480f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:30.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 -- 172.21.15.203:0/176352828 shutdown_connections 2024-02-19T22:14:30.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.451+0000 7fc847fff700 1 -- 172.21.15.203:0/176352828 wait complete. 2024-02-19T22:14:30.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc847fff700 1 Processor -- start 2024-02-19T22:14:30.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc847fff700 1 -- start start 2024-02-19T22:14:30.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc847fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc847fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc848069e20 con 0x7fc8480f62a0 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc846ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc846ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59496/0 (socket says 172.21.15.203:59496) 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.452+0000 7fc846ffd700 1 -- 172.21.15.203:0/3576290332 learned_addr learned my addr 172.21.15.203:0/3576290332 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc846ffd700 1 -- 172.21.15.203:0/3576290332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc83c005040 con 0x7fc8480f62a0 2024-02-19T22:14:30.460 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc846ffd700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc83c009830 tx=0x7fc83c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:30.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc83c009320 con 0x7fc8480f62a0 2024-02-19T22:14:30.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc848069fc0 con 0x7fc8480f62a0 2024-02-19T22:14:30.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc83c013380 con 0x7fc8480f62a0 2024-02-19T22:14:30.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc83c014420 con 0x7fc8480f62a0 2024-02-19T22:14:30.461 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.453+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84806a450 con 0x7fc8480f62a0 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.454+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc83c01e420 con 0x7fc8480f62a0 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.454+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc848059100 con 0x7fc8480f62a0 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.454+0000 7fc82f7fe700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 0x7fc830032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.454+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc83c040e30 con 0x7fc8480f62a0 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.454+0000 7fc8467fc700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 0x7fc830032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:30.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.455+0000 7fc8467fc700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 0x7fc830032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc8380096a0 tx=0x7fc838006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:30.463 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.459+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc83c01cb10 con 0x7fc8480f62a0 2024-02-19T22:14:30.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.666+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc848045ad0 con 0x7fc8480f62a0 2024-02-19T22:14:30.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.667+0000 7fc82f7fe700 1 -- 172.21.15.203:0/3576290332 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc83c004490 con 0x7fc8480f62a0 2024-02-19T22:14:30.669 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:30.669 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:30.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 msgr2=0x7fc830032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 0x7fc830032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fc8380096a0 tx=0x7fc838006b40).stop 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 msgr2=0x7fc84806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fc83c009830 tx=0x7fc83c00b040).stop 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 shutdown_connections 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc83002ff60 0x7fc830032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 --2- 172.21.15.203:0/3576290332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8480f62a0 0x7fc84806b9a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 >> 172.21.15.203:0/3576290332 conn(0x7fc8480f1aa0 msgr2=0x7fc8480f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 shutdown_connections 2024-02-19T22:14:30.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:30.669+0000 7fc847fff700 1 -- 172.21.15.203:0/3576290332 wait complete. 2024-02-19T22:14:30.672 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:29.168334+0000 mgr.smithi111.ftjzov (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:31.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:30.668449+0000 mon.smithi111 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.203:0/3576290332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:32.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:32.534 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:32.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:31.168697+0000 mgr.smithi111.ftjzov (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- 172.21.15.203:0/2505883477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 msgr2=0x7f87f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/2505883477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f40f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f87e4009fb0 tx=0x7f87e4009b30).stop 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- 172.21.15.203:0/2505883477 shutdown_connections 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/2505883477 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f40f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- 172.21.15.203:0/2505883477 >> 172.21.15.203:0/2505883477 conn(0x7f87f40f0b70 msgr2=0x7f87f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- 172.21.15.203:0/2505883477 shutdown_connections 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- 172.21.15.203:0/2505883477 wait complete. 2024-02-19T22:14:34.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 Processor -- start 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.422+0000 7f87fbcf3700 1 -- start start 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87fbcf3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87fbcf3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4060240 con 0x7f87f40f5410 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87f9a8f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87f9a8f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59514/0 (socket says 172.21.15.203:59514) 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87f9a8f700 1 -- 172.21.15.203:0/1000173146 learned_addr learned my addr 172.21.15.203:0/1000173146 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87f9a8f700 1 -- 172.21.15.203:0/1000173146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4005040 con 0x7f87f40f5410 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87f9a8f700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f87e40089f0 tx=0x7f87e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e4009320 con 0x7f87f40f5410 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87e4013380 con 0x7f87f40f5410 2024-02-19T22:14:34.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e401ed80 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4060440 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.423+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4109bd0 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.424+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87e401e420 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.424+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f4045ad0 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.424+0000 7f87ea7fc700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 0x7f87e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.424+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87e4040f00 con 0x7f87f40f5410 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.425+0000 7f87f928e700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 0x7f87e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:34.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.425+0000 7f87f928e700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 0x7f87e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f87f00096a0 tx=0x7f87f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:34.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.429+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87e40095f0 con 0x7f87f40f5410 2024-02-19T22:14:34.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.636+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87f410a3f0 con 0x7f87f40f5410 2024-02-19T22:14:34.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.636+0000 7f87ea7fc700 1 -- 172.21.15.203:0/1000173146 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f87e401cb10 con 0x7f87f40f5410 2024-02-19T22:14:34.638 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:34.638 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 msgr2=0x7f87e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 0x7f87e0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f87f00096a0 tx=0x7f87f0006b40).stop 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 msgr2=0x7f87f4109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f87e40089f0 tx=0x7f87e400b040).stop 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 shutdown_connections 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f87e002ff60 0x7f87e0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 --2- 172.21.15.203:0/1000173146 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f87f40f5410 0x7f87f4109520 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 >> 172.21.15.203:0/1000173146 conn(0x7f87f40f0b70 msgr2=0x7f87f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 shutdown_connections 2024-02-19T22:14:34.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:34.638+0000 7f87fbcf3700 1 -- 172.21.15.203:0/1000173146 wait complete. 2024-02-19T22:14:34.641 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:33.169060+0000 mgr.smithi111.ftjzov (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:35.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:35 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:34.637630+0000 mon.smithi111 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.203:0/1000173146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:36.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:36.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:35.169422+0000 mgr.smithi111.ftjzov (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:38.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.422+0000 7f14cb97c700 1 -- 172.21.15.203:0/1670524383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 msgr2=0x7f14c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:38.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.422+0000 7f14cb97c700 1 --2- 172.21.15.203:0/1670524383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c40f6930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f14b4009fb0 tx=0x7f14b4009b30).stop 2024-02-19T22:14:38.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.422+0000 7f14cb97c700 1 -- 172.21.15.203:0/1670524383 shutdown_connections 2024-02-19T22:14:38.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.422+0000 7f14cb97c700 1 --2- 172.21.15.203:0/1670524383 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c40f6930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.422+0000 7f14cb97c700 1 -- 172.21.15.203:0/1670524383 >> 172.21.15.203:0/1670524383 conn(0x7f14c40f1c70 msgr2=0x7f14c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 -- 172.21.15.203:0/1670524383 shutdown_connections 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 -- 172.21.15.203:0/1670524383 wait complete. 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 Processor -- start 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 -- start start 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14cb97c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c4108ef0 con 0x7f14c40f6510 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14c9718700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14c9718700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39534/0 (socket says 172.21.15.203:39534) 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.423+0000 7f14c9718700 1 -- 172.21.15.203:0/2877017225 learned_addr learned my addr 172.21.15.203:0/2877017225 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:38.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14c9718700 1 -- 172.21.15.203:0/2877017225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b4005040 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14c9718700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f14b40089f0 tx=0x7f14b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b4009320 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41090f0 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14b4013380 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b401ed80 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.424+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c4109580 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.425+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14b401e420 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.425+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c4045ad0 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.425+0000 7f14b9ffb700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.425+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14b4040fc0 con 0x7f14c40f6510 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.426+0000 7f14c8f17700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:38.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.426+0000 7f14c8f17700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 0x7f14b0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f14c00096a0 tx=0x7f14c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:38.432 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.430+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14b40095f0 con 0x7f14c40f6510 2024-02-19T22:14:38.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.637+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14c410a3d0 con 0x7f14c40f6510 2024-02-19T22:14:38.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.637+0000 7f14b9ffb700 1 -- 172.21.15.203:0/2877017225 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14b401cb10 con 0x7f14c40f6510 2024-02-19T22:14:38.639 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:38.639 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 msgr2=0x7f14b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 0x7f14b0032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f14c00096a0 tx=0x7f14c0006b40).stop 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 msgr2=0x7f14c4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f14b40089f0 tx=0x7f14b400b040).stop 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 shutdown_connections 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f14b002ff60 0x7f14b0032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 --2- 172.21.15.203:0/2877017225 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f14c40f6510 0x7f14c4108a00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 >> 172.21.15.203:0/2877017225 conn(0x7f14c40f1c70 msgr2=0x7f14c40f26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 shutdown_connections 2024-02-19T22:14:38.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:38.639+0000 7f14cb97c700 1 -- 172.21.15.203:0/2877017225 wait complete. 2024-02-19T22:14:38.642 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:37.169659+0000 mgr.smithi111.ftjzov (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:39.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:39 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:38.638711+0000 mon.smithi111 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.203:0/2877017225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:40.286 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:40.286 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:39.169869+0000 mgr.smithi111.ftjzov (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 -- 172.21.15.203:0/2921539071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 msgr2=0x7f4bf00f5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/2921539071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00f5810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f4be4009fb0 tx=0x7f4be4009b30).stop 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 -- 172.21.15.203:0/2921539071 shutdown_connections 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/2921539071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00f5810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 -- 172.21.15.203:0/2921539071 >> 172.21.15.203:0/2921539071 conn(0x7f4bf00f0b70 msgr2=0x7f4bf00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 -- 172.21.15.203:0/2921539071 shutdown_connections 2024-02-19T22:14:42.077 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.074+0000 7f4bf78ae700 1 -- 172.21.15.203:0/2921539071 wait complete. 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf78ae700 1 Processor -- start 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf78ae700 1 -- start start 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf78ae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf78ae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bf00fb940 con 0x7f4bf00f53f0 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf564a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf564a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39550/0 (socket says 172.21.15.203:39550) 2024-02-19T22:14:42.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.075+0000 7f4bf564a700 1 -- 172.21.15.203:0/4269080879 learned_addr learned my addr 172.21.15.203:0/4269080879 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:42.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4bf564a700 1 -- 172.21.15.203:0/4269080879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4005040 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4bf564a700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4be4005010 tx=0x7f4be400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be4009320 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4be4013380 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bf00fbae0 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4be4014e50 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.076+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bf00fbf70 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.077+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4be40095f0 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.077+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf0059100 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.077+0000 7f4be1ffb700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 0x7f4bdc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.077+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4be4040f80 con 0x7f4bf00f53f0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.077+0000 7f4bf4e49700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 0x7f4bdc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:42.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.078+0000 7f4bf4e49700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 0x7f4bdc0323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4bec0096a0 tx=0x7f4bec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:42.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.082+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4be4014420 con 0x7f4bf00f53f0 2024-02-19T22:14:42.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.286+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4bf0045ad0 con 0x7f4bf00f53f0 2024-02-19T22:14:42.288 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.287+0000 7f4be1ffb700 1 -- 172.21.15.203:0/4269080879 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4be4010230 con 0x7f4bf00f53f0 2024-02-19T22:14:42.289 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:42.289 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:42.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 msgr2=0x7f4bdc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:42.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 0x7f4bdc0323d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f4bec0096a0 tx=0x7f4bec006b40).stop 2024-02-19T22:14:42.290 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 msgr2=0x7f4bf00fd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f4be4005010 tx=0x7f4be400b040).stop 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 shutdown_connections 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bdc02ff10 0x7f4bdc0323d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 --2- 172.21.15.203:0/4269080879 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4bf00f53f0 0x7f4bf00fd4c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 >> 172.21.15.203:0/4269080879 conn(0x7f4bf00f0b70 msgr2=0x7f4bf005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 shutdown_connections 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:42.288+0000 7f4bf78ae700 1 -- 172.21.15.203:0/4269080879 wait complete. 2024-02-19T22:14:42.291 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:41.170134+0000 mgr.smithi111.ftjzov (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:43.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:42.287974+0000 mon.smithi111 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.203:0/4269080879' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:43.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:43.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:44.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:43.170395+0000 mgr.smithi111.ftjzov (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:45.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 -- 172.21.15.203:0/1728206730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 msgr2=0x7f46b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 --2- 172.21.15.203:0/1728206730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b80f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f46a8009fb0 tx=0x7f46a8009b30).stop 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 -- 172.21.15.203:0/1728206730 shutdown_connections 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 --2- 172.21.15.203:0/1728206730 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b80f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 -- 172.21.15.203:0/1728206730 >> 172.21.15.203:0/1728206730 conn(0x7f46b80f1c70 msgr2=0x7f46b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 -- 172.21.15.203:0/1728206730 shutdown_connections 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.660+0000 7f46c041a700 1 -- 172.21.15.203:0/1728206730 wait complete. 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46c041a700 1 Processor -- start 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46c041a700 1 -- start start 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46c041a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46c041a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b8106d00 con 0x7f46b80f6510 2024-02-19T22:14:45.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46be1b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46be1b6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60652/0 (socket says 172.21.15.203:60652) 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46be1b6700 1 -- 172.21.15.203:0/1893427500 learned_addr learned my addr 172.21.15.203:0/1893427500 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46be1b6700 1 -- 172.21.15.203:0/1893427500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a8005040 con 0x7f46b80f6510 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.661+0000 7f46be1b6700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f46a8005010 tx=0x7f46a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.662+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a8009320 con 0x7f46b80f6510 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.662+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b8106f00 con 0x7f46b80f6510 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.662+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b8107390 con 0x7f46b80f6510 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.662+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f46a8013380 con 0x7f46b80f6510 2024-02-19T22:14:45.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.662+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46a8014df0 con 0x7f46b80f6510 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a0004fa0 con 0x7f46b80f6510 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46a80095f0 con 0x7f46b80f6510 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46aeffd700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 0x7f46a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46a8040e70 con 0x7f46b80f6510 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46bd9b5700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 0x7f46a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:45.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.663+0000 7f46bd9b5700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 0x7f46a40323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:45.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.668+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46a8014420 con 0x7f46b80f6510 2024-02-19T22:14:45.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.875+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46a0005670 con 0x7f46b80f6510 2024-02-19T22:14:45.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.875+0000 7f46aeffd700 1 -- 172.21.15.203:0/1893427500 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46a801d9d0 con 0x7f46b80f6510 2024-02-19T22:14:45.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:45.878 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 msgr2=0x7f46a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 0x7f46a40323d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f46b40096a0 tx=0x7f46b4006b40).stop 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 msgr2=0x7f46b8106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f46a8005010 tx=0x7f46a800b040).stop 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 shutdown_connections 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f46a402ff10 0x7f46a40323d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 --2- 172.21.15.203:0/1893427500 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f46b80f6510 0x7f46b8106810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:45.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 >> 172.21.15.203:0/1893427500 conn(0x7f46b80f1c70 msgr2=0x7f46b80f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:45.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 shutdown_connections 2024-02-19T22:14:45.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:45.877+0000 7f46c041a700 1 -- 172.21.15.203:0/1893427500 wait complete. 2024-02-19T22:14:45.880 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:45.170651+0000 mgr.smithi111.ftjzov (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:45.876852+0000 mon.smithi111 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.203:0/1893427500' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:47.489 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:47.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:48.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:47.170925+0000 mgr.smithi111.ftjzov (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 -- 172.21.15.203:0/2781702307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 msgr2=0x7f63880f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 --2- 172.21.15.203:0/2781702307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63880f8f80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f637c009fb0 tx=0x7f637c009b30).stop 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 -- 172.21.15.203:0/2781702307 shutdown_connections 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 --2- 172.21.15.203:0/2781702307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63880f8f80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 -- 172.21.15.203:0/2781702307 >> 172.21.15.203:0/2781702307 conn(0x7f63880f09a0 msgr2=0x7f63880f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:49.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 -- 172.21.15.203:0/2781702307 shutdown_connections 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.379+0000 7f638da30700 1 -- 172.21.15.203:0/2781702307 wait complete. 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638da30700 1 Processor -- start 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638da30700 1 -- start start 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638da30700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638da30700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6388106ad0 con 0x7f6388060180 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638ca2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638ca2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60664/0 (socket says 172.21.15.203:60664) 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.380+0000 7f638ca2e700 1 -- 172.21.15.203:0/1379038789 learned_addr learned my addr 172.21.15.203:0/1379038789 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f638ca2e700 1 -- 172.21.15.203:0/1379038789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c005040 con 0x7f6388060180 2024-02-19T22:14:49.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f638ca2e700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f637c005010 tx=0x7f637c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:49.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c009320 con 0x7f6388060180 2024-02-19T22:14:49.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6388106cd0 con 0x7f6388060180 2024-02-19T22:14:49.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f637c013380 con 0x7f6388060180 2024-02-19T22:14:49.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f637c014e50 con 0x7f6388060180 2024-02-19T22:14:49.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.381+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6388107160 con 0x7f6388060180 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.382+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f637c0095f0 con 0x7f6388060180 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.382+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6388045ad0 con 0x7f6388060180 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.382+0000 7f63857fa700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 0x7f63780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.382+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f637c040e60 con 0x7f6388060180 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.382+0000 7f6387fff700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 0x7f63780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:49.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.383+0000 7f6387fff700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 0x7f63780323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63740096a0 tx=0x7f6374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:49.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.387+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f637c014420 con 0x7f6388060180 2024-02-19T22:14:49.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.591+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6388107f40 con 0x7f6388060180 2024-02-19T22:14:49.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.592+0000 7f63857fa700 1 -- 172.21.15.203:0/1379038789 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f637c010230 con 0x7f6388060180 2024-02-19T22:14:49.594 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:49.594 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:49.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 msgr2=0x7f63780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 0x7f63780323d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f63740096a0 tx=0x7f6374006b40).stop 2024-02-19T22:14:49.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 msgr2=0x7f63881065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:49.595 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f637c005010 tx=0x7f637c00b040).stop 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 shutdown_connections 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f637802ff10 0x7f63780323d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 --2- 172.21.15.203:0/1379038789 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6388060180 0x7f63881065e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 >> 172.21.15.203:0/1379038789 conn(0x7f63880f09a0 msgr2=0x7f63880f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 shutdown_connections 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:49.593+0000 7f638da30700 1 -- 172.21.15.203:0/1379038789 wait complete. 2024-02-19T22:14:49.596 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:50.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:49.171173+0000 mgr.smithi111.ftjzov (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:50.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:49.592980+0000 mon.smithi111 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.203:0/1379038789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:51.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:51.180 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:52.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:51.171452+0000 mgr.smithi111.ftjzov (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 -- 172.21.15.203:0/1057115843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 msgr2=0x7f27a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 --2- 172.21.15.203:0/1057115843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a80f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f279c009fb0 tx=0x7f279c009b30).stop 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 -- 172.21.15.203:0/1057115843 shutdown_connections 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 --2- 172.21.15.203:0/1057115843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a80f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 -- 172.21.15.203:0/1057115843 >> 172.21.15.203:0/1057115843 conn(0x7f27a80f0b70 msgr2=0x7f27a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 -- 172.21.15.203:0/1057115843 shutdown_connections 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.964+0000 7f27acc85700 1 -- 172.21.15.203:0/1057115843 wait complete. 2024-02-19T22:14:52.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27acc85700 1 Processor -- start 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27acc85700 1 -- start start 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27acc85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27acc85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a8105c00 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27a659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27a659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60674/0 (socket says 172.21.15.203:60674) 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27a659c700 1 -- 172.21.15.203:0/3953528308 learned_addr learned my addr 172.21.15.203:0/3953528308 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27a659c700 1 -- 172.21.15.203:0/3953528308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c005040 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.965+0000 7f27a659c700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f279c005010 tx=0x7f279c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f279c009320 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a8105e00 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f279c013380 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f279c014e50 con 0x7f27a80f5410 2024-02-19T22:14:52.969 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a8106290 con 0x7f27a80f5410 2024-02-19T22:14:52.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.966+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f279c00f410 con 0x7f27a80f5410 2024-02-19T22:14:52.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.967+0000 7f2796ffd700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 0x7f27900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:52.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.967+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f279c042ce0 con 0x7f27a80f5410 2024-02-19T22:14:52.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.967+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a8045ad0 con 0x7f27a80f5410 2024-02-19T22:14:52.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.967+0000 7f27a5d9b700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 0x7f27900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:52.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.967+0000 7f27a5d9b700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 0x7f27900323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f27980096a0 tx=0x7f2798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:52.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:52.971+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f279c0283a0 con 0x7f27a80f5410 2024-02-19T22:14:53.178 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.176+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27a80fb090 con 0x7f27a80f5410 2024-02-19T22:14:53.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.177+0000 7f2796ffd700 1 -- 172.21.15.203:0/3953528308 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f279c012980 con 0x7f27a80f5410 2024-02-19T22:14:53.179 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:53.179 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:53.180 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 msgr2=0x7f27900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 0x7f27900323d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f27980096a0 tx=0x7f2798006b40).stop 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 msgr2=0x7f27a8105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f279c005010 tx=0x7f279c00b040).stop 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 shutdown_connections 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f279002ff10 0x7f27900323d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 --2- 172.21.15.203:0/3953528308 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f27a80f5410 0x7f27a8105710 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 >> 172.21.15.203:0/3953528308 conn(0x7f27a80f0b70 msgr2=0x7f27a80f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 shutdown_connections 2024-02-19T22:14:53.181 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:53.179+0000 7f27acc85700 1 -- 172.21.15.203:0/3953528308 wait complete. 2024-02-19T22:14:53.182 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:53.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:53.178233+0000 mon.smithi111 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.203:0/3953528308' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:53.171670+0000 mgr.smithi111.ftjzov (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:54.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:54.852 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.687+0000 7f9d30fde700 1 -- 172.21.15.203:0/2913657580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 msgr2=0x7f9d2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.687+0000 7f9d30fde700 1 --2- 172.21.15.203:0/2913657580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c0f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f9d1c009fb0 tx=0x7f9d1c009b30).stop 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.687+0000 7f9d30fde700 1 -- 172.21.15.203:0/2913657580 shutdown_connections 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.687+0000 7f9d30fde700 1 --2- 172.21.15.203:0/2913657580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c0f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.687+0000 7f9d30fde700 1 -- 172.21.15.203:0/2913657580 >> 172.21.15.203:0/2913657580 conn(0x7f9d2c0f0b70 msgr2=0x7f9d2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 -- 172.21.15.203:0/2913657580 shutdown_connections 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 -- 172.21.15.203:0/2913657580 wait complete. 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 Processor -- start 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 -- start start 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:56.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d30fde700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2c105bd0 con 0x7f9d2c0f5410 2024-02-19T22:14:56.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d2a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.688+0000 7f9d2a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39648/0 (socket says 172.21.15.203:39648) 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d2a59c700 1 -- 172.21.15.203:0/504840843 learned_addr learned my addr 172.21.15.203:0/504840843 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d2a59c700 1 -- 172.21.15.203:0/504840843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c005040 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d2a59c700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9d1c005010 tx=0x7f9d1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d1c009320 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2c105dd0 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d1c013380 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d1c014e50 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.689+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2c106260 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.690+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d1c0263a0 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.690+0000 7f9d22ffd700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 0x7f9d180323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.690+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2c045ad0 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.690+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d1c041160 con 0x7f9d2c0f5410 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.690+0000 7f9d29d9b700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 0x7f9d180323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:14:56.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.691+0000 7f9d29d9b700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 0x7f9d180323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:14:56.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.695+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d1c012920 con 0x7f9d2c0f5410 2024-02-19T22:14:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:55.172027+0000 mgr.smithi111.ftjzov (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:14:56.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.898+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d2c107110 con 0x7f9d2c0f5410 2024-02-19T22:14:56.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.898+0000 7f9d22ffd700 1 -- 172.21.15.203:0/504840843 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d1c012330 con 0x7f9d2c0f5410 2024-02-19T22:14:56.901 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:14:56.901 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 msgr2=0x7f9d180323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 0x7f9d180323d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f9d140096a0 tx=0x7f9d14006b40).stop 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 msgr2=0x7f9d2c1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f9d1c005010 tx=0x7f9d1c00b040).stop 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 shutdown_connections 2024-02-19T22:14:56.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9d1802ff10 0x7f9d180323d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 --2- 172.21.15.203:0/504840843 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d2c0f5410 0x7f9d2c1056e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:14:56.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 >> 172.21.15.203:0/504840843 conn(0x7f9d2c0f0b70 msgr2=0x7f9d2c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:14:56.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 shutdown_connections 2024-02-19T22:14:56.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:14:56.900+0000 7f9d30fde700 1 -- 172.21.15.203:0/504840843 wait complete. 2024-02-19T22:14:56.904 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:14:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:14:56.899796+0000 mon.smithi111 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.203:0/504840843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:14:58.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:14:58.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:14:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:14:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:57.172228+0000 mgr.smithi111.ftjzov (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:00.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.253+0000 7f51815fa700 1 -- 172.21.15.203:0/531360303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 msgr2=0x7f517c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.253+0000 7f51815fa700 1 --2- 172.21.15.203:0/531360303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c0f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f516c009fb0 tx=0x7f516c009b30).stop 2024-02-19T22:15:00.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.253+0000 7f51815fa700 1 -- 172.21.15.203:0/531360303 shutdown_connections 2024-02-19T22:15:00.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.253+0000 7f51815fa700 1 --2- 172.21.15.203:0/531360303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c0f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.253+0000 7f51815fa700 1 -- 172.21.15.203:0/531360303 >> 172.21.15.203:0/531360303 conn(0x7f517c0f0b70 msgr2=0x7f517c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 -- 172.21.15.203:0/531360303 shutdown_connections 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 -- 172.21.15.203:0/531360303 wait complete. 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 Processor -- start 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 -- start start 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.254+0000 7f51815fa700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c106d00 con 0x7f517c0f5410 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f517affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f517affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39664/0 (socket says 172.21.15.203:39664) 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f517affd700 1 -- 172.21.15.203:0/2420420866 learned_addr learned my addr 172.21.15.203:0/2420420866 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f517affd700 1 -- 172.21.15.203:0/2420420866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516c005040 con 0x7f517c0f5410 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f517affd700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f516c005010 tx=0x7f516c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:00.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516c009320 con 0x7f517c0f5410 2024-02-19T22:15:00.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517c106f00 con 0x7f517c0f5410 2024-02-19T22:15:00.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f516c013380 con 0x7f517c0f5410 2024-02-19T22:15:00.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516c014e50 con 0x7f517c0f5410 2024-02-19T22:15:00.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.255+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c107390 con 0x7f517c0f5410 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.256+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f516c014420 con 0x7f517c0f5410 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.256+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c045ad0 con 0x7f517c0f5410 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.256+0000 7f51637fe700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 0x7f51640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.256+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f516c040f70 con 0x7f517c0f5410 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.256+0000 7f517a7fc700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 0x7f51640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:00.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.257+0000 7f517a7fc700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 0x7f51640323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f51700096a0 tx=0x7f5170006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:00.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.262+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f516c0095f0 con 0x7f517c0f5410 2024-02-19T22:15:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.467+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f517c0fafa0 con 0x7f517c0f5410 2024-02-19T22:15:00.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.467+0000 7f51637fe700 1 -- 172.21.15.203:0/2420420866 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f516c01db70 con 0x7f517c0f5410 2024-02-19T22:15:00.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:00.470 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 msgr2=0x7f51640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 0x7f51640323d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f51700096a0 tx=0x7f5170006b40).stop 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 msgr2=0x7f517c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f516c005010 tx=0x7f516c00b040).stop 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 shutdown_connections 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f516402ff10 0x7f51640323d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 --2- 172.21.15.203:0/2420420866 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f517c0f5410 0x7f517c106810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 >> 172.21.15.203:0/2420420866 conn(0x7f517c0f0b70 msgr2=0x7f517c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:00.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 shutdown_connections 2024-02-19T22:15:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:00.470+0000 7f51815fa700 1 -- 172.21.15.203:0/2420420866 wait complete. 2024-02-19T22:15:00.472 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:14:59.172601+0000 mgr.smithi111.ftjzov (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:01.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:00.468746+0000 mon.smithi111 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.203:0/2420420866' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:02.014 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:02.014 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:01.172937+0000 mgr.smithi111.ftjzov (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.896+0000 7fc7786c9700 1 -- 172.21.15.203:0/1210069864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 msgr2=0x7fc7700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.896+0000 7fc7786c9700 1 --2- 172.21.15.203:0/1210069864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7700f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc764009fb0 tx=0x7fc764009b30).stop 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 -- 172.21.15.203:0/1210069864 shutdown_connections 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 --2- 172.21.15.203:0/1210069864 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7700f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 -- 172.21.15.203:0/1210069864 >> 172.21.15.203:0/1210069864 conn(0x7fc7700f0b70 msgr2=0x7fc7700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 -- 172.21.15.203:0/1210069864 shutdown_connections 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 -- 172.21.15.203:0/1210069864 wait complete. 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 Processor -- start 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.897+0000 7fc7786c9700 1 -- start start 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc7786c9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc7786c9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770105bd0 con 0x7fc7700f5410 2024-02-19T22:15:03.899 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc776465700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:03.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc776465700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39686/0 (socket says 172.21.15.203:39686) 2024-02-19T22:15:03.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc776465700 1 -- 172.21.15.203:0/4198350707 learned_addr learned my addr 172.21.15.203:0/4198350707 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:03.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc776465700 1 -- 172.21.15.203:0/4198350707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc764005040 con 0x7fc7700f5410 2024-02-19T22:15:03.900 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc776465700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc764005010 tx=0x7fc76400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764009320 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc770105dd0 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc764013380 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc764014e50 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.898+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc770106260 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.899+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7640095f0 con 0x7fc7700f5410 2024-02-19T22:15:03.901 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.899+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc770045ad0 con 0x7fc7700f5410 2024-02-19T22:15:03.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.899+0000 7fc762ffd700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 0x7fc75c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:03.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.899+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc764040e70 con 0x7fc7700f5410 2024-02-19T22:15:03.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.900+0000 7fc775c64700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 0x7fc75c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:03.902 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.900+0000 7fc775c64700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 0x7fc75c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc76c0096a0 tx=0x7fc76c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:03.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:03.904+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc764014420 con 0x7fc7700f5410 2024-02-19T22:15:04.111 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.110+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc770107110 con 0x7fc7700f5410 2024-02-19T22:15:04.111 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.111+0000 7fc762ffd700 1 -- 172.21.15.203:0/4198350707 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc764010230 con 0x7fc7700f5410 2024-02-19T22:15:04.112 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:04.112 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 msgr2=0x7fc75c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 0x7fc75c0323d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fc76c0096a0 tx=0x7fc76c006b40).stop 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 msgr2=0x7fc7701056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc764005010 tx=0x7fc76400b040).stop 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 shutdown_connections 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.112+0000 7fc7786c9700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc75c02ff10 0x7fc75c0323d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.113+0000 7fc7786c9700 1 --2- 172.21.15.203:0/4198350707 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7700f5410 0x7fc7701056e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.113+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 >> 172.21.15.203:0/4198350707 conn(0x7fc7700f0b70 msgr2=0x7fc7700f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:04.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.113+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 shutdown_connections 2024-02-19T22:15:04.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:04.113+0000 7fc7786c9700 1 -- 172.21.15.203:0/4198350707 wait complete. 2024-02-19T22:15:04.115 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:03.173321+0000 mgr.smithi111.ftjzov (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:04.111233+0000 mon.smithi111 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.203:0/4198350707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:05.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:05.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:05.173643+0000 mgr.smithi111.ftjzov (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- 172.21.15.203:0/3440955568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 msgr2=0x7f582c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 --2- 172.21.15.203:0/3440955568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c0f55c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5820009fb0 tx=0x7f5820009b30).stop 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- 172.21.15.203:0/3440955568 shutdown_connections 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 --2- 172.21.15.203:0/3440955568 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c0f55c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- 172.21.15.203:0/3440955568 >> 172.21.15.203:0/3440955568 conn(0x7f582c0f09a0 msgr2=0x7f582c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- 172.21.15.203:0/3440955568 shutdown_connections 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- 172.21.15.203:0/3440955568 wait complete. 2024-02-19T22:15:07.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 Processor -- start 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.652+0000 7f5831eca700 1 -- start start 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5831eca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5831eca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c060280 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5830ec8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5830ec8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38146/0 (socket says 172.21.15.203:38146) 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5830ec8700 1 -- 172.21.15.203:0/2900429213 learned_addr learned my addr 172.21.15.203:0/2900429213 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5830ec8700 1 -- 172.21.15.203:0/2900429213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820005040 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f5830ec8700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f5820009830 tx=0x7f582000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.653+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5820009320 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5820013380 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c060480 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5820014420 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c1099c0 con 0x7f582c0f51a0 2024-02-19T22:15:07.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f582001e420 con 0x7f582c0f51a0 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.654+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c045ad0 con 0x7f582c0f51a0 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.655+0000 7f58297fa700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 0x7f581c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.655+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5820040e20 con 0x7f582c0f51a0 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.655+0000 7f582bfff700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 0x7f581c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.655+0000 7f582bfff700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 0x7f581c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f58180096a0 tx=0x7f5818006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:07.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.659+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f582001c970 con 0x7f582c0f51a0 2024-02-19T22:15:07.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.865+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f582c0fad30 con 0x7f582c0f51a0 2024-02-19T22:15:07.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.866+0000 7f58297fa700 1 -- 172.21.15.203:0/2900429213 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f582002c2d0 con 0x7f582c0f51a0 2024-02-19T22:15:07.868 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:07.868 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:07.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 msgr2=0x7f581c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 0x7f581c032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f58180096a0 tx=0x7f5818006b40).stop 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 msgr2=0x7f582c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f5820009830 tx=0x7f582000b040).stop 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 shutdown_connections 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f581c02ff60 0x7f581c032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 --2- 172.21.15.203:0/2900429213 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f582c0f51a0 0x7f582c109290 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 >> 172.21.15.203:0/2900429213 conn(0x7f582c0f09a0 msgr2=0x7f582c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 shutdown_connections 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:07.868+0000 7f5831eca700 1 -- 172.21.15.203:0/2900429213 wait complete. 2024-02-19T22:15:07.870 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:07.173840+0000 mgr.smithi111.ftjzov (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:07.867334+0000 mon.smithi111 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.203:0/2900429213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:09.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:09.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:09.174179+0000 mgr.smithi111.ftjzov (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 -- 172.21.15.203:0/814007870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 msgr2=0x7f52000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 --2- 172.21.15.203:0/814007870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52000f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f51e8009fb0 tx=0x7f51e8009b30).stop 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 -- 172.21.15.203:0/814007870 shutdown_connections 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 --2- 172.21.15.203:0/814007870 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52000f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 -- 172.21.15.203:0/814007870 >> 172.21.15.203:0/814007870 conn(0x7f52000f0b70 msgr2=0x7f52000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 -- 172.21.15.203:0/814007870 shutdown_connections 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.368+0000 7f5204a41700 1 -- 172.21.15.203:0/814007870 wait complete. 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.369+0000 7f5204a41700 1 Processor -- start 2024-02-19T22:15:11.371 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.369+0000 7f5204a41700 1 -- start start 2024-02-19T22:15:11.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.369+0000 7f5204a41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:11.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.369+0000 7f5204a41700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5200105bd0 con 0x7f52000f5410 2024-02-19T22:15:11.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51fe59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38152/0 (socket says 172.21.15.203:38152) 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51fe59c700 1 -- 172.21.15.203:0/3417517219 learned_addr learned my addr 172.21.15.203:0/3417517219 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51fe59c700 1 -- 172.21.15.203:0/3417517219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8005040 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51fe59c700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f51e80089f0 tx=0x7f51e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e8009320 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f51e8013380 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5200105dd0 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f51e801ed80 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.370+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5200106260 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.371+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51e801e420 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.371+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5200045ad0 con 0x7f52000f5410 2024-02-19T22:15:11.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.371+0000 7f51f6ffd700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 0x7f51ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:11.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.371+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51e8041090 con 0x7f52000f5410 2024-02-19T22:15:11.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.371+0000 7f51fdd9b700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 0x7f51ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:11.375 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.372+0000 7f51fdd9b700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 0x7f51ec032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f51f00096a0 tx=0x7f51f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:11.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.376+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51e80095f0 con 0x7f52000f5410 2024-02-19T22:15:11.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.581+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5200107110 con 0x7f52000f5410 2024-02-19T22:15:11.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.582+0000 7f51f6ffd700 1 -- 172.21.15.203:0/3417517219 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f51e8004490 con 0x7f52000f5410 2024-02-19T22:15:11.584 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:11.584 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:11.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 msgr2=0x7f51ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 0x7f51ec032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f51f00096a0 tx=0x7f51f0006b40).stop 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 msgr2=0x7f52001056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f51e80089f0 tx=0x7f51e800b040).stop 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 shutdown_connections 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51ec02ff60 0x7f51ec032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 --2- 172.21.15.203:0/3417517219 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f52000f5410 0x7f52001056e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 >> 172.21.15.203:0/3417517219 conn(0x7f52000f0b70 msgr2=0x7f52000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 shutdown_connections 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:11.583+0000 7f5204a41700 1 -- 172.21.15.203:0/3417517219 wait complete. 2024-02-19T22:15:11.586 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:11.174526+0000 mgr.smithi111.ftjzov (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:12.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:11.582835+0000 mon.smithi111 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.203:0/3417517219' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:13.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:13.254 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:13.174790+0000 mgr.smithi111.ftjzov (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:15.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 -- 172.21.15.203:0/3910745257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 msgr2=0x7fbecc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 --2- 172.21.15.203:0/3910745257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc0f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fbec0009fb0 tx=0x7fbec0009b30).stop 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 -- 172.21.15.203:0/3910745257 shutdown_connections 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 --2- 172.21.15.203:0/3910745257 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc0f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 -- 172.21.15.203:0/3910745257 >> 172.21.15.203:0/3910745257 conn(0x7fbecc0f1c70 msgr2=0x7fbecc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 -- 172.21.15.203:0/3910745257 shutdown_connections 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.139+0000 7fbed09e6700 1 -- 172.21.15.203:0/3910745257 wait complete. 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbed09e6700 1 Processor -- start 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbed09e6700 1 -- start start 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbed09e6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbed09e6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbecc106cd0 con 0x7fbecc0f6510 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbeca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbeca59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38160/0 (socket says 172.21.15.203:38160) 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbeca59c700 1 -- 172.21.15.203:0/1297301734 learned_addr learned my addr 172.21.15.203:0/1297301734 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbeca59c700 1 -- 172.21.15.203:0/1297301734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0005040 con 0x7fbecc0f6510 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbeca59c700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbec0005010 tx=0x7fbec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.140+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0009320 con 0x7fbecc0f6510 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbecc106ed0 con 0x7fbecc0f6510 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbec0013380 con 0x7fbecc0f6510 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbec0014e50 con 0x7fbecc0f6510 2024-02-19T22:15:15.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbecc107360 con 0x7fbecc0f6510 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbec00263a0 con 0x7fbecc0f6510 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbebaffd700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 0x7fbeb40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.141+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbec0040fa0 con 0x7fbecc0f6510 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.142+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbecc059100 con 0x7fbecc0f6510 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.142+0000 7fbec9d9b700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 0x7fbeb40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:15.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.142+0000 7fbec9d9b700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 0x7fbeb40323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fbebc0096a0 tx=0x7fbebc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:15.148 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.146+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbec0012920 con 0x7fbecc0f6510 2024-02-19T22:15:15.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.351+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbecc0fc0a0 con 0x7fbecc0f6510 2024-02-19T22:15:15.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.352+0000 7fbebaffd700 1 -- 172.21.15.203:0/1297301734 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbec002b020 con 0x7fbecc0f6510 2024-02-19T22:15:15.354 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:15.354 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:15.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 msgr2=0x7fbeb40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 0x7fbeb40323d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fbebc0096a0 tx=0x7fbebc006b40).stop 2024-02-19T22:15:15.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 msgr2=0x7fbecc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:15.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbec0005010 tx=0x7fbec000b040).stop 2024-02-19T22:15:15.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 shutdown_connections 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbeb402ff10 0x7fbeb40323d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 --2- 172.21.15.203:0/1297301734 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbecc0f6510 0x7fbecc1067e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.353+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 >> 172.21.15.203:0/1297301734 conn(0x7fbecc0f1c70 msgr2=0x7fbecc0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.354+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 shutdown_connections 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:15.354+0000 7fbed09e6700 1 -- 172.21.15.203:0/1297301734 wait complete. 2024-02-19T22:15:15.356 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:15.175123+0000 mgr.smithi111.ftjzov (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:15.353198+0000 mon.smithi111 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.203:0/1297301734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:17.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:17.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:17.175421+0000 mgr.smithi111.ftjzov (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:17.297036+0000 mon.smithi111 (mon.0) 141 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:15:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:17.299564+0000 mon.smithi111 (mon.0) 142 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 -- 172.21.15.203:0/1381579155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 msgr2=0x7f1aac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/1381579155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac0f6930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f1aa0009fb0 tx=0x7f1aa0009b30).stop 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 -- 172.21.15.203:0/1381579155 shutdown_connections 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/1381579155 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac0f6930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 -- 172.21.15.203:0/1381579155 >> 172.21.15.203:0/1381579155 conn(0x7f1aac0f1c70 msgr2=0x7f1aac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 -- 172.21.15.203:0/1381579155 shutdown_connections 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.950+0000 7f1ab0eec700 1 -- 172.21.15.203:0/1381579155 wait complete. 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1ab0eec700 1 Processor -- start 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1ab0eec700 1 -- start start 2024-02-19T22:15:18.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1ab0eec700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:18.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1ab0eec700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aac106cd0 con 0x7f1aac0f6510 2024-02-19T22:15:18.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1aaa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:18.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1aaa59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53976/0 (socket says 172.21.15.203:53976) 2024-02-19T22:15:18.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.951+0000 7f1aaa59c700 1 -- 172.21.15.203:0/2759303425 learned_addr learned my addr 172.21.15.203:0/2759303425 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1aaa59c700 1 -- 172.21.15.203:0/2759303425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa0005040 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1aaa59c700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1aa0005010 tx=0x7f1aa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa0009320 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aac106ed0 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1aa0013380 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1aa0014e50 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.952+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aac107360 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1aa00095f0 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aac045ad0 con 0x7f1aac0f6510 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1a9affd700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 0x7f1a940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:18.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1aa0040e90 con 0x7f1aac0f6510 2024-02-19T22:15:18.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1aa9d9b700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 0x7f1a940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:18.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.953+0000 7f1aa9d9b700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 0x7f1a940323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1a9c0096a0 tx=0x7f1a9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:18.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:18.958+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1aa0014420 con 0x7f1aac0f6510 2024-02-19T22:15:19.163 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.162+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1aac108210 con 0x7f1aac0f6510 2024-02-19T22:15:19.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.162+0000 7f1a9affd700 1 -- 172.21.15.203:0/2759303425 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1aa0010230 con 0x7f1aac0f6510 2024-02-19T22:15:19.164 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:19.164 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 msgr2=0x7f1a940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 0x7f1a940323d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f1a9c0096a0 tx=0x7f1a9c006b40).stop 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 msgr2=0x7f1aac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f1aa0005010 tx=0x7f1aa000b040).stop 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 shutdown_connections 2024-02-19T22:15:19.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1a9402ff10 0x7f1a940323d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 --2- 172.21.15.203:0/2759303425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1aac0f6510 0x7f1aac1067e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:19.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 >> 172.21.15.203:0/2759303425 conn(0x7f1aac0f1c70 msgr2=0x7f1aac0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:19.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 shutdown_connections 2024-02-19T22:15:19.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:19.164+0000 7f1ab0eec700 1 -- 172.21.15.203:0/2759303425 wait complete. 2024-02-19T22:15:19.167 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:19.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:19.163654+0000 mon.smithi111 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.203:0/2759303425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:19.175671+0000 mgr.smithi111.ftjzov (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:20.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:20.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:22.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:21.175965+0000 mgr.smithi111.ftjzov (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 -- 172.21.15.203:0/1905305493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 msgr2=0x7fddd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 --2- 172.21.15.203:0/1905305493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd00f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fddc0009fb0 tx=0x7fddc0009b30).stop 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 -- 172.21.15.203:0/1905305493 shutdown_connections 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 --2- 172.21.15.203:0/1905305493 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd00f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 -- 172.21.15.203:0/1905305493 >> 172.21.15.203:0/1905305493 conn(0x7fddd00f0b70 msgr2=0x7fddd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 -- 172.21.15.203:0/1905305493 shutdown_connections 2024-02-19T22:15:22.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.779+0000 7fddd4a71700 1 -- 172.21.15.203:0/1905305493 wait complete. 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddd4a71700 1 Processor -- start 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddd4a71700 1 -- start start 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddd4a71700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddd4a71700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd0060220 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddce59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54002/0 (socket says 172.21.15.203:54002) 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddce59c700 1 -- 172.21.15.203:0/729622854 learned_addr learned my addr 172.21.15.203:0/729622854 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.780+0000 7fddce59c700 1 -- 172.21.15.203:0/729622854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc0005040 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddce59c700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fddc0005010 tx=0x7fddc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc0009320 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd0060420 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fddc0013380 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fddc0014e50 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.781+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd0109bb0 con 0x7fddd00f5410 2024-02-19T22:15:22.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fddc00095f0 con 0x7fddd00f5410 2024-02-19T22:15:22.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd0059100 con 0x7fddd00f5410 2024-02-19T22:15:22.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddbeffd700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:22.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fddc0040fa0 con 0x7fddd00f5410 2024-02-19T22:15:22.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddcdd9b700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:22.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.782+0000 7fddcdd9b700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 0x7fddb80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fddc40096a0 tx=0x7fddc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:22.788 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.786+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fddc0014420 con 0x7fddd00f5410 2024-02-19T22:15:22.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.989+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fddd0045ad0 con 0x7fddd00f5410 2024-02-19T22:15:22.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.990+0000 7fddbeffd700 1 -- 172.21.15.203:0/729622854 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fddc0010230 con 0x7fddd00f5410 2024-02-19T22:15:22.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:22.992 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:22.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.991+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 msgr2=0x7fddb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.991+0000 7fddd4a71700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 0x7fddb80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fddc40096a0 tx=0x7fddc4006b40).stop 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.991+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 msgr2=0x7fddd0109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.991+0000 7fddd4a71700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fddc0005010 tx=0x7fddc000b040).stop 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 shutdown_connections 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fddb802ff10 0x7fddb80323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 --2- 172.21.15.203:0/729622854 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fddd00f5410 0x7fddd0109500 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 >> 172.21.15.203:0/729622854 conn(0x7fddd00f0b70 msgr2=0x7fddd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 shutdown_connections 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:22.992+0000 7fddd4a71700 1 -- 172.21.15.203:0/729622854 wait complete. 2024-02-19T22:15:22.994 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:22.991186+0000 mon.smithi111 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.203:0/729622854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:24.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:23.176222+0000 mgr.smithi111.ftjzov (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:24.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:24.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:26.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:25.176543+0000 mgr.smithi111.ftjzov (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:26.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- 172.21.15.203:0/2003108963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 msgr2=0x7f39b00f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 --2- 172.21.15.203:0/2003108963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b00f8fc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f39ac009fb0 tx=0x7f39ac009b30).stop 2024-02-19T22:15:26.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- 172.21.15.203:0/2003108963 shutdown_connections 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 --2- 172.21.15.203:0/2003108963 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b00f8fc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- 172.21.15.203:0/2003108963 >> 172.21.15.203:0/2003108963 conn(0x7f39b00f09a0 msgr2=0x7f39b00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- 172.21.15.203:0/2003108963 shutdown_connections 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- 172.21.15.203:0/2003108963 wait complete. 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 Processor -- start 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- start start 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.588+0000 7f39b614d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0106ae0 con 0x7f39b0060140 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b514b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b514b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44426/0 (socket says 172.21.15.203:44426) 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b514b700 1 -- 172.21.15.203:0/3867401976 learned_addr learned my addr 172.21.15.203:0/3867401976 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b514b700 1 -- 172.21.15.203:0/3867401976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac005040 con 0x7f39b0060140 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b514b700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f39ac005010 tx=0x7f39ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:26.592 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ac009320 con 0x7f39b0060140 2024-02-19T22:15:26.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0106ce0 con 0x7f39b0060140 2024-02-19T22:15:26.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39ac013380 con 0x7f39b0060140 2024-02-19T22:15:26.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39ac014e50 con 0x7f39b0060140 2024-02-19T22:15:26.593 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.589+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0107170 con 0x7f39b0060140 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.590+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f39ac0095f0 con 0x7f39b0060140 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.590+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0045ad0 con 0x7f39b0060140 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.590+0000 7f39a5ffb700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 0x7f399c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.590+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39ac040e70 con 0x7f39b0060140 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.591+0000 7f39b494a700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 0x7f399c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:26.594 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.591+0000 7f39b494a700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 0x7f399c0323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f39a00096a0 tx=0x7f39a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:26.597 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.595+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39ac014420 con 0x7f39b0060140 2024-02-19T22:15:26.800 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.798+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39b0108020 con 0x7f39b0060140 2024-02-19T22:15:26.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.799+0000 7f39a5ffb700 1 -- 172.21.15.203:0/3867401976 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f39ac010230 con 0x7f39b0060140 2024-02-19T22:15:26.801 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:26.801 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 msgr2=0x7f399c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 0x7f399c0323d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f39a00096a0 tx=0x7f39a0006b40).stop 2024-02-19T22:15:26.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 msgr2=0x7f39b01065f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f39ac005010 tx=0x7f39ac00b040).stop 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 shutdown_connections 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f399c02ff10 0x7f399c0323d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 --2- 172.21.15.203:0/3867401976 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f39b0060140 0x7f39b01065f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 >> 172.21.15.203:0/3867401976 conn(0x7f39b00f09a0 msgr2=0x7f39b00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 shutdown_connections 2024-02-19T22:15:26.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:26.800+0000 7f39b614d700 1 -- 172.21.15.203:0/3867401976 wait complete. 2024-02-19T22:15:26.804 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:27.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:26.800127+0000 mon.smithi111 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.203:0/3867401976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:28.390 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:28.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:28.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:27.176882+0000 mgr.smithi111.ftjzov (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:30.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 -- 172.21.15.203:0/1337670127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 msgr2=0x7f6b0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 --2- 172.21.15.203:0/1337670127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f6afc009fb0 tx=0x7f6afc009b30).stop 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 -- 172.21.15.203:0/1337670127 shutdown_connections 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 --2- 172.21.15.203:0/1337670127 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 -- 172.21.15.203:0/1337670127 >> 172.21.15.203:0/1337670127 conn(0x7f6b0c0f0b70 msgr2=0x7f6b0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 -- 172.21.15.203:0/1337670127 shutdown_connections 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 -- 172.21.15.203:0/1337670127 wait complete. 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.211+0000 7f6b134c3700 1 Processor -- start 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b134c3700 1 -- start start 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b134c3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:30.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b134c3700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0c109500 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b1125f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b1125f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44440/0 (socket says 172.21.15.203:44440) 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b1125f700 1 -- 172.21.15.203:0/76643811 learned_addr learned my addr 172.21.15.203:0/76643811 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b1125f700 1 -- 172.21.15.203:0/76643811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc005040 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b1125f700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6afc005010 tx=0x7f6afc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.212+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6afc009320 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.213+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b0c109640 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.213+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6afc013380 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.213+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6afc014e50 con 0x7f6b0c0f5410 2024-02-19T22:15:30.216 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.213+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0c109b30 con 0x7f6b0c0f5410 2024-02-19T22:15:30.217 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.213+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6afc0095f0 con 0x7f6b0c0f5410 2024-02-19T22:15:30.217 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.214+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b0c045ad0 con 0x7f6b0c0f5410 2024-02-19T22:15:30.217 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.214+0000 7f6b01ffb700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 0x7f6af80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:30.217 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.214+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6afc040fa0 con 0x7f6b0c0f5410 2024-02-19T22:15:30.217 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.214+0000 7f6b10a5e700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 0x7f6af80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:30.218 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.214+0000 7f6b10a5e700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 0x7f6af80323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6b080096a0 tx=0x7f6b08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:30.220 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.218+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6afc0268c0 con 0x7f6b0c0f5410 2024-02-19T22:15:30.425 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.423+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b0c10a360 con 0x7f6b0c0f5410 2024-02-19T22:15:30.426 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.425+0000 7f6b01ffb700 1 -- 172.21.15.203:0/76643811 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6afc010230 con 0x7f6b0c0f5410 2024-02-19T22:15:30.426 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:30.426 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:30.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 msgr2=0x7f6af80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 0x7f6af80323d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f6b080096a0 tx=0x7f6b08006b40).stop 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 msgr2=0x7f6b0c0601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6afc005010 tx=0x7f6afc00b040).stop 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 shutdown_connections 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6af802ff10 0x7f6af80323d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 --2- 172.21.15.203:0/76643811 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6b0c0f5410 0x7f6b0c0601b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 >> 172.21.15.203:0/76643811 conn(0x7f6b0c0f0b70 msgr2=0x7f6b0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 shutdown_connections 2024-02-19T22:15:30.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:30.426+0000 7f6b134c3700 1 -- 172.21.15.203:0/76643811 wait complete. 2024-02-19T22:15:30.429 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:30.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:29.177246+0000 mgr.smithi111.ftjzov (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:31.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:30.425271+0000 mon.smithi111 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.203:0/76643811' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:32.057 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:32.058 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:32.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:31.177570+0000 mgr.smithi111.ftjzov (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 -- 172.21.15.203:0/3931896136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 msgr2=0x7f9d100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 --2- 172.21.15.203:0/3931896136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d100f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f9d00009fb0 tx=0x7f9d00009b30).stop 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 -- 172.21.15.203:0/3931896136 shutdown_connections 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 --2- 172.21.15.203:0/3931896136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d100f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 -- 172.21.15.203:0/3931896136 >> 172.21.15.203:0/3931896136 conn(0x7f9d100f09a0 msgr2=0x7f9d100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 -- 172.21.15.203:0/3931896136 shutdown_connections 2024-02-19T22:15:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 -- 172.21.15.203:0/3931896136 wait complete. 2024-02-19T22:15:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.847+0000 7f9d16103700 1 Processor -- start 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d16103700 1 -- start start 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d16103700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d16103700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d101059c0 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d15101700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d15101700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44452/0 (socket says 172.21.15.203:44452) 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d15101700 1 -- 172.21.15.203:0/2972178862 learned_addr learned my addr 172.21.15.203:0/2972178862 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d15101700 1 -- 172.21.15.203:0/2972178862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00005040 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d15101700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9d000089f0 tx=0x7f9d0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.848+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d00009320 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.849+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d10105bc0 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.849+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9d00013380 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.849+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d00014e20 con 0x7f9d100f51a0 2024-02-19T22:15:33.851 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.849+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d10106050 con 0x7f9d100f51a0 2024-02-19T22:15:33.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.849+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d000095f0 con 0x7f9d100f51a0 2024-02-19T22:15:33.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.850+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d10045ad0 con 0x7f9d100f51a0 2024-02-19T22:15:33.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.850+0000 7f9d05ffb700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 0x7f9cfc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:33.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.850+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d00040f50 con 0x7f9d100f51a0 2024-02-19T22:15:33.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.850+0000 7f9d14900700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 0x7f9cfc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:33.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.850+0000 7f9d14900700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 0x7f9cfc0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9d0c0096a0 tx=0x7f9d0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:33.856 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:33.855+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d00014420 con 0x7f9d100f51a0 2024-02-19T22:15:34.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.060+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d10106f00 con 0x7f9d100f51a0 2024-02-19T22:15:34.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.060+0000 7f9d05ffb700 1 -- 172.21.15.203:0/2972178862 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d0001d980 con 0x7f9d100f51a0 2024-02-19T22:15:34.062 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:34.063 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:34.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 msgr2=0x7f9cfc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:34.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 0x7f9cfc0323d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f9d0c0096a0 tx=0x7f9d0c006b40).stop 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 msgr2=0x7f9d101054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f9d000089f0 tx=0x7f9d0000b040).stop 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 shutdown_connections 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cfc02ff10 0x7f9cfc0323d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 --2- 172.21.15.203:0/2972178862 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d100f51a0 0x7f9d101054d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 >> 172.21.15.203:0/2972178862 conn(0x7f9d100f09a0 msgr2=0x7f9d100f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 shutdown_connections 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:34.062+0000 7f9d16103700 1 -- 172.21.15.203:0/2972178862 wait complete. 2024-02-19T22:15:34.065 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:34.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:33.177914+0000 mgr.smithi111.ftjzov (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:34.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:34.061552+0000 mon.smithi111 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.203:0/2972178862' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:35.757 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:35.758 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:36.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:35.178260+0000 mgr.smithi111.ftjzov (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:37.614 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/405965703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 msgr2=0x7fc9bc0f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/405965703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f4b60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fc9b4009960 tx=0x7fc9b4004d10).stop 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/405965703 shutdown_connections 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/405965703 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f4b60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/405965703 >> 172.21.15.203:0/405965703 conn(0x7fc9bc0f0260 msgr2=0x7fc9bc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/405965703 shutdown_connections 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/405965703 wait complete. 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.596+0000 7fc9c2e9c700 1 Processor -- start 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c2e9c700 1 -- start start 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c2e9c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c2e9c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0f5770 con 0x7fc9bc0f4740 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c0c38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:37.615 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c0c38700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37194/0 (socket says 172.21.15.203:37194) 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c0c38700 1 -- 172.21.15.203:0/3229363634 learned_addr learned my addr 172.21.15.203:0/3229363634 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c0c38700 1 -- 172.21.15.203:0/3229363634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009640 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.597+0000 7fc9c0c38700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc9b4004b70 tx=0x7fc9b4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b4018070 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc0f5910 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9b400b970 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc9b401c560 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc0f5da0 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc9b4013630 con 0x7fc9bc0f4740 2024-02-19T22:15:37.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.598+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc059100 con 0x7fc9bc0f4740 2024-02-19T22:15:37.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.599+0000 7fc9b97fa700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:37.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.599+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc9b4041320 con 0x7fc9bc0f4740 2024-02-19T22:15:37.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.599+0000 7fc9bbfff700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:37.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.599+0000 7fc9bbfff700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 0x7fc9a8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc9ac0096a0 tx=0x7fc9ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:37.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.604+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc9b4012970 con 0x7fc9bc0f4740 2024-02-19T22:15:37.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.809+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc9bc0fc080 con 0x7fc9bc0f4740 2024-02-19T22:15:37.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.809+0000 7fc9b97fa700 1 -- 172.21.15.203:0/3229363634 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc9b4012380 con 0x7fc9bc0f4740 2024-02-19T22:15:37.811 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:37.811 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:37.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 msgr2=0x7fc9a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 0x7fc9a8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fc9ac0096a0 tx=0x7fc9ac006b40).stop 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 msgr2=0x7fc9bc0f72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fc9b4004b70 tx=0x7fc9b4004d10).stop 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 shutdown_connections 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc9a802ff60 0x7fc9a8032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 --2- 172.21.15.203:0/3229363634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc9bc0f4740 0x7fc9bc0f72f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 >> 172.21.15.203:0/3229363634 conn(0x7fc9bc0f0260 msgr2=0x7fc9bc0f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 shutdown_connections 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:37.811+0000 7fc9c2e9c700 1 -- 172.21.15.203:0/3229363634 wait complete. 2024-02-19T22:15:37.813 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:38.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:37.178493+0000 mgr.smithi111.ftjzov (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:38.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:37.810457+0000 mon.smithi111 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.203:0/3229363634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:39.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:39.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:40.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:39.178825+0000 mgr.smithi111.ftjzov (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 -- 172.21.15.203:0/493909108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 msgr2=0x7f95880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 --2- 172.21.15.203:0/493909108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95880f5830 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f9570009fb0 tx=0x7f9570009b30).stop 2024-02-19T22:15:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 -- 172.21.15.203:0/493909108 shutdown_connections 2024-02-19T22:15:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 --2- 172.21.15.203:0/493909108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95880f5830 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 -- 172.21.15.203:0/493909108 >> 172.21.15.203:0/493909108 conn(0x7f95880f0b70 msgr2=0x7f95880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 -- 172.21.15.203:0/493909108 shutdown_connections 2024-02-19T22:15:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.169+0000 7f958e1ee700 1 -- 172.21.15.203:0/493909108 wait complete. 2024-02-19T22:15:41.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.170+0000 7f958e1ee700 1 Processor -- start 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.170+0000 7f958e1ee700 1 -- start start 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.170+0000 7f958e1ee700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.170+0000 7f958e1ee700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9588105bd0 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f95877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f95877fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37220/0 (socket says 172.21.15.203:37220) 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f95877fe700 1 -- 172.21.15.203:0/2425256895 learned_addr learned my addr 172.21.15.203:0/2425256895 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f95877fe700 1 -- 172.21.15.203:0/2425256895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9570005040 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f95877fe700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9570005010 tx=0x7f957000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9570009320 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9588105dd0 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9570013380 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.171+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9570014e50 con 0x7f95880f5410 2024-02-19T22:15:41.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.172+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9588106260 con 0x7f95880f5410 2024-02-19T22:15:41.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.172+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95700095f0 con 0x7f95880f5410 2024-02-19T22:15:41.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.172+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588045ad0 con 0x7f95880f5410 2024-02-19T22:15:41.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.172+0000 7f957ffff700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 0x7f95740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:41.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.172+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9570040f80 con 0x7f95880f5410 2024-02-19T22:15:41.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.173+0000 7f9586ffd700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 0x7f95740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:41.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.173+0000 7f9586ffd700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 0x7f95740323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f95780096a0 tx=0x7f9578006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:41.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.177+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9570014420 con 0x7f95880f5410 2024-02-19T22:15:41.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.381+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9588107110 con 0x7f95880f5410 2024-02-19T22:15:41.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.382+0000 7f957ffff700 1 -- 172.21.15.203:0/2425256895 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9570010230 con 0x7f95880f5410 2024-02-19T22:15:41.383 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:41.383 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:41.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 msgr2=0x7f95740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 0x7f95740323d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f95780096a0 tx=0x7f9578006b40).stop 2024-02-19T22:15:41.384 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 msgr2=0x7f95881056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f9570005010 tx=0x7f957000b040).stop 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 shutdown_connections 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.383+0000 7f958e1ee700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f957402ff10 0x7f95740323d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.384+0000 7f958e1ee700 1 --2- 172.21.15.203:0/2425256895 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95880f5410 0x7f95881056e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.384+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 >> 172.21.15.203:0/2425256895 conn(0x7f95880f0b70 msgr2=0x7f95880f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.384+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 shutdown_connections 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:41.384+0000 7f958e1ee700 1 -- 172.21.15.203:0/2425256895 wait complete. 2024-02-19T22:15:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:41.179085+0000 mgr.smithi111.ftjzov (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:41.382227+0000 mon.smithi111 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.203:0/2425256895' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:43.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:43.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:44.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:43.179430+0000 mgr.smithi111.ftjzov (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 -- 172.21.15.203:0/119177081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 msgr2=0x7fa8200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 --2- 172.21.15.203:0/119177081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8200f5830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa814009fb0 tx=0x7fa814009b30).stop 2024-02-19T22:15:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 -- 172.21.15.203:0/119177081 shutdown_connections 2024-02-19T22:15:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 --2- 172.21.15.203:0/119177081 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8200f5830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 -- 172.21.15.203:0/119177081 >> 172.21.15.203:0/119177081 conn(0x7fa8200f0b70 msgr2=0x7fa8200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 -- 172.21.15.203:0/119177081 shutdown_connections 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.014+0000 7fa826852700 1 -- 172.21.15.203:0/119177081 wait complete. 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa826852700 1 Processor -- start 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa826852700 1 -- start start 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa826852700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa826852700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820106cd0 con 0x7fa8200f5410 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa81ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa81ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37236/0 (socket says 172.21.15.203:37236) 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.015+0000 7fa81ffff700 1 -- 172.21.15.203:0/3173649391 learned_addr learned my addr 172.21.15.203:0/3173649391 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa81ffff700 1 -- 172.21.15.203:0/3173649391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa814005040 con 0x7fa8200f5410 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa81ffff700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa814005010 tx=0x7fa81400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa814009320 con 0x7fa8200f5410 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa820106ed0 con 0x7fa8200f5410 2024-02-19T22:15:45.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa814013380 con 0x7fa8200f5410 2024-02-19T22:15:45.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa814014e50 con 0x7fa8200f5410 2024-02-19T22:15:45.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.016+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa820107360 con 0x7fa8200f5410 2024-02-19T22:15:45.019 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa814014420 con 0x7fa8200f5410 2024-02-19T22:15:45.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820045ad0 con 0x7fa8200f5410 2024-02-19T22:15:45.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa81cff9700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 0x7fa8080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:45.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa814040f00 con 0x7fa8200f5410 2024-02-19T22:15:45.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa81f7fe700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 0x7fa8080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:45.020 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.017+0000 7fa81f7fe700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 0x7fa8080323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa8100096a0 tx=0x7fa810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:45.022 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.022+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8140095f0 con 0x7fa8200f5410 2024-02-19T22:15:45.226 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.225+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8200fafa0 con 0x7fa8200f5410 2024-02-19T22:15:45.226 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.225+0000 7fa81cff9700 1 -- 172.21.15.203:0/3173649391 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa81401db70 con 0x7fa8200f5410 2024-02-19T22:15:45.227 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:45.227 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 msgr2=0x7fa8080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 0x7fa8080323d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fa8100096a0 tx=0x7fa810006b40).stop 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 msgr2=0x7fa8201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fa814005010 tx=0x7fa81400b040).stop 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 shutdown_connections 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa80802ff10 0x7fa8080323d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 --2- 172.21.15.203:0/3173649391 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa8200f5410 0x7fa8201067e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 >> 172.21.15.203:0/3173649391 conn(0x7fa8200f0b70 msgr2=0x7fa82005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 shutdown_connections 2024-02-19T22:15:45.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:45.227+0000 7fa826852700 1 -- 172.21.15.203:0/3173649391 wait complete. 2024-02-19T22:15:45.229 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:45.179743+0000 mgr.smithi111.ftjzov (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:45.225659+0000 mon.smithi111 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.203:0/3173649391' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:46.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:46.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:48.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:47.180101+0000 mgr.smithi111.ftjzov (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:48.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 -- 172.21.15.203:0/2152552368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 msgr2=0x7f3ec40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 --2- 172.21.15.203:0/2152552368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f90f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f3eb4009fb0 tx=0x7f3eb4009b30).stop 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 -- 172.21.15.203:0/2152552368 shutdown_connections 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 --2- 172.21.15.203:0/2152552368 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f90f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 -- 172.21.15.203:0/2152552368 >> 172.21.15.203:0/2152552368 conn(0x7f3ec40f0b70 msgr2=0x7f3ec40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 -- 172.21.15.203:0/2152552368 shutdown_connections 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.885+0000 7f3ecc020700 1 -- 172.21.15.203:0/2152552368 wait complete. 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ecc020700 1 Processor -- start 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ecc020700 1 -- start start 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ecc020700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:48.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ecc020700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40f4ff0 con 0x7f3ec4060280 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ec9dbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ec9dbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49302/0 (socket says 172.21.15.203:49302) 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ec9dbc700 1 -- 172.21.15.203:0/17756494 learned_addr learned my addr 172.21.15.203:0/17756494 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.886+0000 7f3ec9dbc700 1 -- 172.21.15.203:0/17756494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb4005040 con 0x7f3ec4060280 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3ec9dbc700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3eb40089f0 tx=0x7f3eb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb4009320 con 0x7f3ec4060280 2024-02-19T22:15:48.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec40f5190 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3eb4013380 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3eb4014e20 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.887+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec40f5620 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3eb40263a0 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3eba7fc700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 0x7f3eb00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3eb4040eb0 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec4059100 con 0x7f3ec4060280 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3ec95bb700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 0x7f3eb00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:48.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.888+0000 7f3ec95bb700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 0x7f3eb00323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3ec00096a0 tx=0x7f3ec0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:48.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:48.893+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3eb4012920 con 0x7f3ec4060280 2024-02-19T22:15:49.099 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.097+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ec40f5d20 con 0x7f3ec4060280 2024-02-19T22:15:49.100 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.098+0000 7f3eba7fc700 1 -- 172.21.15.203:0/17756494 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3eb400f340 con 0x7f3ec4060280 2024-02-19T22:15:49.100 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:49.100 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 msgr2=0x7f3eb00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 0x7f3eb00323d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f3ec00096a0 tx=0x7f3ec0006b40).stop 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 msgr2=0x7f3ec40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3eb40089f0 tx=0x7f3eb400b040).stop 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 shutdown_connections 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3eb002ff10 0x7f3eb00323d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 --2- 172.21.15.203:0/17756494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ec4060280 0x7f3ec40f6b70 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 >> 172.21.15.203:0/17756494 conn(0x7f3ec40f0b70 msgr2=0x7f3ec40f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 shutdown_connections 2024-02-19T22:15:49.102 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:49.100+0000 7f3ecc020700 1 -- 172.21.15.203:0/17756494 wait complete. 2024-02-19T22:15:49.103 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:49.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:49 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:49.099270+0000 mon.smithi111 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.203:0/17756494' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:50.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:49.180384+0000 mgr.smithi111.ftjzov (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:50.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:50.802 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:52.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:51.180604+0000 mgr.smithi111.ftjzov (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:52.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 -- 172.21.15.203:0/3648575434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 msgr2=0x7fba580f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/3648575434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba580f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fba4c009fb0 tx=0x7fba4c009b30).stop 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 -- 172.21.15.203:0/3648575434 shutdown_connections 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/3648575434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba580f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 -- 172.21.15.203:0/3648575434 >> 172.21.15.203:0/3648575434 conn(0x7fba580f1c70 msgr2=0x7fba580f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 -- 172.21.15.203:0/3648575434 shutdown_connections 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 -- 172.21.15.203:0/3648575434 wait complete. 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.780+0000 7fba5c8c8700 1 Processor -- start 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5c8c8700 1 -- start start 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5c8c8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:52.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5c8c8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba58106cd0 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49326/0 (socket says 172.21.15.203:49326) 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5659c700 1 -- 172.21.15.203:0/152935063 learned_addr learned my addr 172.21.15.203:0/152935063 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5659c700 1 -- 172.21.15.203:0/152935063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c005040 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5659c700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fba4c0089f0 tx=0x7fba4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba4c009320 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba58106ed0 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.781+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba4c013380 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba4c01ed80 con 0x7fba580f6510 2024-02-19T22:15:52.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba58107360 con 0x7fba580f6510 2024-02-19T22:15:52.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba4c01e420 con 0x7fba580f6510 2024-02-19T22:15:52.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba58045ad0 con 0x7fba580f6510 2024-02-19T22:15:52.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba46ffd700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 0x7fba40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:52.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.782+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba4c041150 con 0x7fba580f6510 2024-02-19T22:15:52.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.783+0000 7fba55d9b700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 0x7fba40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:52.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.783+0000 7fba55d9b700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 0x7fba40032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fba480096a0 tx=0x7fba48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:52.789 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.787+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba4c0095f0 con 0x7fba580f6510 2024-02-19T22:15:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.990+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba58108210 con 0x7fba580f6510 2024-02-19T22:15:52.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.990+0000 7fba46ffd700 1 -- 172.21.15.203:0/152935063 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba4c004490 con 0x7fba580f6510 2024-02-19T22:15:52.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:52.992 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 msgr2=0x7fba40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 0x7fba40032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fba480096a0 tx=0x7fba48006b40).stop 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 msgr2=0x7fba581067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fba4c0089f0 tx=0x7fba4c00b040).stop 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 shutdown_connections 2024-02-19T22:15:52.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba4002ff60 0x7fba40032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 --2- 172.21.15.203:0/152935063 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba580f6510 0x7fba581067e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 >> 172.21.15.203:0/152935063 conn(0x7fba580f1c70 msgr2=0x7fba580f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 shutdown_connections 2024-02-19T22:15:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:52.992+0000 7fba5c8c8700 1 -- 172.21.15.203:0/152935063 wait complete. 2024-02-19T22:15:52.995 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:53.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:52.991541+0000 mon.smithi111 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.203:0/152935063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:54.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:53.180953+0000 mgr.smithi111.ftjzov (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:54.609 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:54.609 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 -- 172.21.15.203:0/2920265387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 msgr2=0x7f96580f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 --2- 172.21.15.203:0/2920265387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f96580f5750 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f9648009fb0 tx=0x7f9648009b30).stop 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 -- 172.21.15.203:0/2920265387 shutdown_connections 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 --2- 172.21.15.203:0/2920265387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f96580f5750 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 -- 172.21.15.203:0/2920265387 >> 172.21.15.203:0/2920265387 conn(0x7f96580f0a90 msgr2=0x7f96580f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 -- 172.21.15.203:0/2920265387 shutdown_connections 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.423+0000 7f965f50f700 1 -- 172.21.15.203:0/2920265387 wait complete. 2024-02-19T22:15:56.427 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965f50f700 1 Processor -- start 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965f50f700 1 -- start start 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965f50f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965f50f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9658107590 con 0x7f96580f5330 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965d2ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965d2ab700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34748/0 (socket says 172.21.15.203:34748) 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.424+0000 7f965d2ab700 1 -- 172.21.15.203:0/1324389906 learned_addr learned my addr 172.21.15.203:0/1324389906 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f965d2ab700 1 -- 172.21.15.203:0/1324389906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648005040 con 0x7f96580f5330 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f965d2ab700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9648005010 tx=0x7f964800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9648009320 con 0x7f96580f5330 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9658060220 con 0x7f96580f5330 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9648013380 con 0x7f96580f5330 2024-02-19T22:15:56.428 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9648014e50 con 0x7f96580f5330 2024-02-19T22:15:56.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.425+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965810c090 con 0x7f96580f5330 2024-02-19T22:15:56.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.426+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96480263a0 con 0x7f96580f5330 2024-02-19T22:15:56.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.426+0000 7f964dffb700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 0x7f96440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:15:56.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.426+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9658045ad0 con 0x7f96580f5330 2024-02-19T22:15:56.429 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.426+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9648041290 con 0x7f96580f5330 2024-02-19T22:15:56.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.427+0000 7f965caaa700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 0x7f96440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:15:56.430 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.427+0000 7f965caaa700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 0x7f96440323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f96540096a0 tx=0x7f9654006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:15:56.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.431+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9648012330 con 0x7f96580f5330 2024-02-19T22:15:56.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:55.181279+0000 mgr.smithi111.ftjzov (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:15:56.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.637+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f965805ea50 con 0x7f96580f5330 2024-02-19T22:15:56.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.637+0000 7f964dffb700 1 -- 172.21.15.203:0/1324389906 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f964800f340 con 0x7f96580f5330 2024-02-19T22:15:56.639 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:15:56.639 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 msgr2=0x7f96440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 0x7f96440323d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f96540096a0 tx=0x7f9654006b40).stop 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 msgr2=0x7f965810bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f9648005010 tx=0x7f964800b040).stop 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 shutdown_connections 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f964402ff10 0x7f96440323d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 --2- 172.21.15.203:0/1324389906 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f96580f5330 0x7f965810bba0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 >> 172.21.15.203:0/1324389906 conn(0x7f96580f0a90 msgr2=0x7f96580f1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:15:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 shutdown_connections 2024-02-19T22:15:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:15:56.639+0000 7f965f50f700 1 -- 172.21.15.203:0/1324389906 wait complete. 2024-02-19T22:15:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:15:57.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:15:56.638680+0000 mon.smithi111 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.203:0/1324389906' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:15:58.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:15:58.294 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:15:58.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:15:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:57.181495+0000 mgr.smithi111.ftjzov (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:00.111 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 -- 172.21.15.203:0/2600113434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 msgr2=0x7f48e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 --2- 172.21.15.203:0/2600113434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e00f5830 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f48d4009fb0 tx=0x7f48d4009b30).stop 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 -- 172.21.15.203:0/2600113434 shutdown_connections 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 --2- 172.21.15.203:0/2600113434 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e00f5830 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 -- 172.21.15.203:0/2600113434 >> 172.21.15.203:0/2600113434 conn(0x7f48e00f0b70 msgr2=0x7f48e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 -- 172.21.15.203:0/2600113434 shutdown_connections 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.108+0000 7f48e487b700 1 -- 172.21.15.203:0/2600113434 wait complete. 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48e487b700 1 Processor -- start 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48e487b700 1 -- start start 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48e487b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48e487b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e0060280 con 0x7f48e00f5410 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48de59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48de59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34770/0 (socket says 172.21.15.203:34770) 2024-02-19T22:16:00.112 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.109+0000 7f48de59c700 1 -- 172.21.15.203:0/3285636286 learned_addr learned my addr 172.21.15.203:0/3285636286 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:00.113 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48de59c700 1 -- 172.21.15.203:0/3285636286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4005040 con 0x7f48e00f5410 2024-02-19T22:16:00.113 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48de59c700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f48d4005010 tx=0x7f48d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d4009320 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e0060480 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48d4013380 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48d4014e50 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.110+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e0109bd0 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.111+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48d40263a0 con 0x7f48e00f5410 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.111+0000 7f48ceffd700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 0x7f48c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:00.114 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.111+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48d4040f80 con 0x7f48e00f5410 2024-02-19T22:16:00.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.111+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e0045ad0 con 0x7f48e00f5410 2024-02-19T22:16:00.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.111+0000 7f48ddd9b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 0x7f48c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:00.115 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.112+0000 7f48ddd9b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 0x7f48c80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f48d00096a0 tx=0x7f48d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:00.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.116+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48d4012920 con 0x7f48e00f5410 2024-02-19T22:16:00.323 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.322+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48e00fafa0 con 0x7f48e00f5410 2024-02-19T22:16:00.324 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.322+0000 7f48ceffd700 1 -- 172.21.15.203:0/3285636286 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48d402b020 con 0x7f48e00f5410 2024-02-19T22:16:00.324 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:00.324 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 msgr2=0x7f48c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 0x7f48c80323d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f48d00096a0 tx=0x7f48d0006b40).stop 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 msgr2=0x7f48e01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f48d4005010 tx=0x7f48d400b040).stop 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 shutdown_connections 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48c802ff10 0x7f48c80323d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 --2- 172.21.15.203:0/3285636286 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f48e00f5410 0x7f48e01094a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 >> 172.21.15.203:0/3285636286 conn(0x7f48e00f0b70 msgr2=0x7f48e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:00.326 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 shutdown_connections 2024-02-19T22:16:00.327 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:00.324+0000 7f48e487b700 1 -- 172.21.15.203:0/3285636286 wait complete. 2024-02-19T22:16:00.327 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:15:59.181831+0000 mgr.smithi111.ftjzov (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:01.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:00.323505+0000 mon.smithi111 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.203:0/3285636286' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:02.012 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:02.012 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:01.182128+0000 mgr.smithi111.ftjzov (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 -- 172.21.15.203:0/1153626863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 msgr2=0x7f95380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 --2- 172.21.15.203:0/1153626863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95380f55c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f9528009fb0 tx=0x7f9528009b30).stop 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 -- 172.21.15.203:0/1153626863 shutdown_connections 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 --2- 172.21.15.203:0/1153626863 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95380f55c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 -- 172.21.15.203:0/1153626863 >> 172.21.15.203:0/1153626863 conn(0x7f95380f09a0 msgr2=0x7f95380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 -- 172.21.15.203:0/1153626863 shutdown_connections 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.824+0000 7f9537fff700 1 -- 172.21.15.203:0/1153626863 wait complete. 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9537fff700 1 Processor -- start 2024-02-19T22:16:03.877 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9537fff700 1 -- start start 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9537fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9537fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9538106ad0 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9536ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34786/0 (socket says 172.21.15.203:34786) 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9536ffd700 1 -- 172.21.15.203:0/2048500189 learned_addr learned my addr 172.21.15.203:0/2048500189 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9536ffd700 1 -- 172.21.15.203:0/2048500189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9528005040 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.825+0000 7f9536ffd700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f95280089f0 tx=0x7f952800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9528009320 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9538106cd0 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9528013380 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9528014e20 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9538107160 con 0x7f95380f51a0 2024-02-19T22:16:03.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95280095f0 con 0x7f95380f51a0 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.826+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9538045ad0 con 0x7f95380f51a0 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.827+0000 7f951f7fe700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 0x7f95200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.827+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9528040e90 con 0x7f95380f51a0 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.827+0000 7f95367fc700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 0x7f95200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.827+0000 7f95367fc700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 0x7f95200323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f952c0096a0 tx=0x7f952c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:03.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:03.832+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9528014420 con 0x7f95380f51a0 2024-02-19T22:16:04.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.037+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9538108010 con 0x7f95380f51a0 2024-02-19T22:16:04.039 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.038+0000 7f951f7fe700 1 -- 172.21.15.203:0/2048500189 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f952801d9d0 con 0x7f95380f51a0 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:04.040 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 msgr2=0x7f95200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 0x7f95200323d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f952c0096a0 tx=0x7f952c006b40).stop 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 msgr2=0x7f95381065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f95280089f0 tx=0x7f952800b040).stop 2024-02-19T22:16:04.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 shutdown_connections 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f952002ff10 0x7f95200323d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 --2- 172.21.15.203:0/2048500189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f95380f51a0 0x7f95381065e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 >> 172.21.15.203:0/2048500189 conn(0x7f95380f09a0 msgr2=0x7f95380f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 shutdown_connections 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:04.039+0000 7f9537fff700 1 -- 172.21.15.203:0/2048500189 wait complete. 2024-02-19T22:16:04.042 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:04.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:03.182547+0000 mgr.smithi111.ftjzov (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:04.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:04.039016+0000 mon.smithi111 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.203:0/2048500189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:05.675 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:05.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:06.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:05.182892+0000 mgr.smithi111.ftjzov (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 -- 172.21.15.203:0/3063152009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 msgr2=0x7f9a5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 --2- 172.21.15.203:0/3063152009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c0f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9a50009fb0 tx=0x7f9a50009b30).stop 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 -- 172.21.15.203:0/3063152009 shutdown_connections 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 --2- 172.21.15.203:0/3063152009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c0f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 -- 172.21.15.203:0/3063152009 >> 172.21.15.203:0/3063152009 conn(0x7f9a5c0f1c70 msgr2=0x7f9a5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 -- 172.21.15.203:0/3063152009 shutdown_connections 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.463+0000 7f9a61235700 1 -- 172.21.15.203:0/3063152009 wait complete. 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a61235700 1 Processor -- start 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a61235700 1 -- start start 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a61235700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:07.467 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a61235700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5c106cd0 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a5ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a5ad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:48968/0 (socket says 172.21.15.203:48968) 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a5ad9d700 1 -- 172.21.15.203:0/770363565 learned_addr learned my addr 172.21.15.203:0/770363565 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a5ad9d700 1 -- 172.21.15.203:0/770363565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a50005040 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.464+0000 7f9a5ad9d700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9a50005010 tx=0x7f9a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a50009320 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5c106ed0 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a50013380 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a50014e50 con 0x7f9a5c0f6510 2024-02-19T22:16:07.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5c107360 con 0x7f9a5c0f6510 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a50014420 con 0x7f9a5c0f6510 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.465+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5c045ad0 con 0x7f9a5c0f6510 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.466+0000 7f9a437fe700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 0x7f9a440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.466+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a50040f00 con 0x7f9a5c0f6510 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.466+0000 7f9a5a59c700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 0x7f9a440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:07.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.466+0000 7f9a5a59c700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 0x7f9a440323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9a4c0096a0 tx=0x7f9a4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:07.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.470+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a50014790 con 0x7f9a5c0f6510 2024-02-19T22:16:07.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.677+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a5c108210 con 0x7f9a5c0f6510 2024-02-19T22:16:07.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.677+0000 7f9a437fe700 1 -- 172.21.15.203:0/770363565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9a5001d380 con 0x7f9a5c0f6510 2024-02-19T22:16:07.679 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:07.680 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 msgr2=0x7f9a440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 0x7f9a440323d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9a4c0096a0 tx=0x7f9a4c006b40).stop 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 msgr2=0x7f9a5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9a50005010 tx=0x7f9a5000b040).stop 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 shutdown_connections 2024-02-19T22:16:07.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9a4402ff10 0x7f9a440323d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 --2- 172.21.15.203:0/770363565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9a5c0f6510 0x7f9a5c1067e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:07.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 >> 172.21.15.203:0/770363565 conn(0x7f9a5c0f1c70 msgr2=0x7f9a5c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:07.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 shutdown_connections 2024-02-19T22:16:07.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:07.679+0000 7f9a61235700 1 -- 172.21.15.203:0/770363565 wait complete. 2024-02-19T22:16:07.682 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:08.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:07.183296+0000 mgr.smithi111.ftjzov (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:08.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:07.678610+0000 mon.smithi111 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.203:0/770363565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:09.388 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:09.388 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:10.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:09.183623+0000 mgr.smithi111.ftjzov (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:11.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 -- 172.21.15.203:0/712493184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 msgr2=0x7f62800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 --2- 172.21.15.203:0/712493184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f62800f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f627c009fb0 tx=0x7f627c009b30).stop 2024-02-19T22:16:11.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 -- 172.21.15.203:0/712493184 shutdown_connections 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 --2- 172.21.15.203:0/712493184 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f62800f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 -- 172.21.15.203:0/712493184 >> 172.21.15.203:0/712493184 conn(0x7f62800f0b70 msgr2=0x7f62800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 -- 172.21.15.203:0/712493184 shutdown_connections 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 -- 172.21.15.203:0/712493184 wait complete. 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.352+0000 7f6287f2f700 1 Processor -- start 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6287f2f700 1 -- start start 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6287f2f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6287f2f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62801072f0 con 0x7f62800f5410 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6285ccb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6285ccb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:48982/0 (socket says 172.21.15.203:48982) 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6285ccb700 1 -- 172.21.15.203:0/220073634 learned_addr learned my addr 172.21.15.203:0/220073634 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6285ccb700 1 -- 172.21.15.203:0/220073634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c005040 con 0x7f62800f5410 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6285ccb700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f627c005010 tx=0x7f627c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:11.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627c009320 con 0x7f62800f5410 2024-02-19T22:16:11.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6280060220 con 0x7f62800f5410 2024-02-19T22:16:11.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f627c013380 con 0x7f62800f5410 2024-02-19T22:16:11.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.353+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627c014e50 con 0x7f62800f5410 2024-02-19T22:16:11.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.354+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628010bdf0 con 0x7f62800f5410 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.354+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f627c0095f0 con 0x7f62800f5410 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.354+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6280045ad0 con 0x7f62800f5410 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.354+0000 7f62767fc700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 0x7f626c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.354+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f627c040e90 con 0x7f62800f5410 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.355+0000 7f62854ca700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 0x7f626c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:11.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.355+0000 7f62854ca700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 0x7f626c0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:11.361 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.359+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f627c014420 con 0x7f62800f5410 2024-02-19T22:16:11.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.564+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f628010c5b0 con 0x7f62800f5410 2024-02-19T22:16:11.566 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.564+0000 7f62767fc700 1 -- 172.21.15.203:0/220073634 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f627c010230 con 0x7f62800f5410 2024-02-19T22:16:11.567 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:11.567 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 msgr2=0x7f626c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 0x7f626c0323d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f62700096a0 tx=0x7f6270006b40).stop 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 msgr2=0x7f628010b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f627c005010 tx=0x7f627c00b040).stop 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 shutdown_connections 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f626c02ff10 0x7f626c0323d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 --2- 172.21.15.203:0/220073634 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62800f5410 0x7f628010b900 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:11.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 >> 172.21.15.203:0/220073634 conn(0x7f62800f0b70 msgr2=0x7f62800f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:11.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 shutdown_connections 2024-02-19T22:16:11.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:11.566+0000 7f6287f2f700 1 -- 172.21.15.203:0/220073634 wait complete. 2024-02-19T22:16:11.569 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:11.183901+0000 mgr.smithi111.ftjzov (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:11.565699+0000 mon.smithi111 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.203:0/220073634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:13.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:13.202 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:14.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:13.184228+0000 mgr.smithi111.ftjzov (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 -- 172.21.15.203:0/3916626932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 msgr2=0x7f24dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 --2- 172.21.15.203:0/3916626932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc0f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f24cc009fb0 tx=0x7f24cc009b30).stop 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 -- 172.21.15.203:0/3916626932 shutdown_connections 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 --2- 172.21.15.203:0/3916626932 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc0f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 -- 172.21.15.203:0/3916626932 >> 172.21.15.203:0/3916626932 conn(0x7f24dc0f0b70 msgr2=0x7f24dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 -- 172.21.15.203:0/3916626932 shutdown_connections 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.011+0000 7f24e420b700 1 -- 172.21.15.203:0/3916626932 wait complete. 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e420b700 1 Processor -- start 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e420b700 1 -- start start 2024-02-19T22:16:15.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e420b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e420b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc106cf0 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e1fa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e1fa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49006/0 (socket says 172.21.15.203:49006) 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e1fa7700 1 -- 172.21.15.203:0/1679158997 learned_addr learned my addr 172.21.15.203:0/1679158997 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e1fa7700 1 -- 172.21.15.203:0/1679158997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc005040 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.012+0000 7f24e1fa7700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f24cc0089f0 tx=0x7f24cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc009320 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc106ef0 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24cc013380 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc01ed80 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc107380 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24cc01e420 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.013+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc045ad0 con 0x7f24dc0f5410 2024-02-19T22:16:15.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.014+0000 7f24d27fc700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 0x7f24c8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:15.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.014+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24cc041090 con 0x7f24dc0f5410 2024-02-19T22:16:15.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.014+0000 7f24e17a6700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 0x7f24c8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:15.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.014+0000 7f24e17a6700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 0x7f24c8032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f24d80096a0 tx=0x7f24d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:15.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.018+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24cc0095f0 con 0x7f24dc0f5410 2024-02-19T22:16:15.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.223+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24dc108230 con 0x7f24dc0f5410 2024-02-19T22:16:15.225 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24d27fc700 1 -- 172.21.15.203:0/1679158997 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f24cc004490 con 0x7f24dc0f5410 2024-02-19T22:16:15.226 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:15.226 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:15.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 msgr2=0x7f24c8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 0x7f24c8032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f24d80096a0 tx=0x7f24d8006b40).stop 2024-02-19T22:16:15.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 msgr2=0x7f24dc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:15.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f24cc0089f0 tx=0x7f24cc00b040).stop 2024-02-19T22:16:15.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 shutdown_connections 2024-02-19T22:16:15.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f24c802ff60 0x7f24c8032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 --2- 172.21.15.203:0/1679158997 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f24dc0f5410 0x7f24dc106800 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:15.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 >> 172.21.15.203:0/1679158997 conn(0x7f24dc0f0b70 msgr2=0x7f24dc0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:15.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 shutdown_connections 2024-02-19T22:16:15.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:15.225+0000 7f24e420b700 1 -- 172.21.15.203:0/1679158997 wait complete. 2024-02-19T22:16:15.229 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:16.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:15.184613+0000 mgr.smithi111.ftjzov (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:16.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:15.225020+0000 mon.smithi111 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.203:0/1679158997' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:16.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:16.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:17.184915+0000 mgr.smithi111.ftjzov (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:17.298100+0000 mon.smithi111 (mon.0) 159 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:16:18.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:17.300653+0000 mon.smithi111 (mon.0) 160 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 -- 172.21.15.203:0/3506728320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 msgr2=0x7f55680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 --2- 172.21.15.203:0/3506728320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55680f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f5558009fb0 tx=0x7f5558009b30).stop 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 -- 172.21.15.203:0/3506728320 shutdown_connections 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 --2- 172.21.15.203:0/3506728320 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55680f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 -- 172.21.15.203:0/3506728320 >> 172.21.15.203:0/3506728320 conn(0x7f55680f0b70 msgr2=0x7f55680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:18.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 -- 172.21.15.203:0/3506728320 shutdown_connections 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.715+0000 7f556fccd700 1 -- 172.21.15.203:0/3506728320 wait complete. 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.716+0000 7f556fccd700 1 Processor -- start 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.716+0000 7f556fccd700 1 -- start start 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.716+0000 7f556fccd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.716+0000 7f556fccd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5568105bd0 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556da69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556da69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34386/0 (socket says 172.21.15.203:34386) 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556da69700 1 -- 172.21.15.203:0/1724677645 learned_addr learned my addr 172.21.15.203:0/1724677645 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556da69700 1 -- 172.21.15.203:0/1724677645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5558005040 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556da69700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5558005010 tx=0x7f555800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5558009320 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5568105dd0 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5558013380 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5558014e50 con 0x7f55680f5410 2024-02-19T22:16:18.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.717+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5568106260 con 0x7f55680f5410 2024-02-19T22:16:18.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.718+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55580263a0 con 0x7f55680f5410 2024-02-19T22:16:18.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.718+0000 7f555e7fc700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 0x7f55540323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:18.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.718+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5558040ea0 con 0x7f55680f5410 2024-02-19T22:16:18.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.718+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5568045ad0 con 0x7f55680f5410 2024-02-19T22:16:18.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.718+0000 7f556d268700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 0x7f55540323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:18.721 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.719+0000 7f556d268700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 0x7f55540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f55640096a0 tx=0x7f5564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:18.724 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.723+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5558012920 con 0x7f55680f5410 2024-02-19T22:16:18.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.928+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f55680fb090 con 0x7f55680f5410 2024-02-19T22:16:18.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.929+0000 7f555e7fc700 1 -- 172.21.15.203:0/1724677645 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5558012330 con 0x7f55680f5410 2024-02-19T22:16:18.930 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:18.930 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:18.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 msgr2=0x7f55540323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 0x7f55540323d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f55640096a0 tx=0x7f5564006b40).stop 2024-02-19T22:16:18.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 msgr2=0x7f55681056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:18.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f5558005010 tx=0x7f555800b040).stop 2024-02-19T22:16:18.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 shutdown_connections 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f555402ff10 0x7f55540323d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 --2- 172.21.15.203:0/1724677645 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f55680f5410 0x7f55681056e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 >> 172.21.15.203:0/1724677645 conn(0x7f55680f0b70 msgr2=0x7f55680f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 shutdown_connections 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:18.931+0000 7f556fccd700 1 -- 172.21.15.203:0/1724677645 wait complete. 2024-02-19T22:16:18.933 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:19.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:18.929398+0000 mon.smithi111 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.203:0/1724677645' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:20.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:19.185215+0000 mgr.smithi111.ftjzov (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:20.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:20.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:22.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:21.185455+0000 mgr.smithi111.ftjzov (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 -- 172.21.15.203:0/3934980039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 msgr2=0x7f35380f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 --2- 172.21.15.203:0/3934980039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f55c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f3528009fb0 tx=0x7f3528009b30).stop 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 -- 172.21.15.203:0/3934980039 shutdown_connections 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 --2- 172.21.15.203:0/3934980039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f55c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 -- 172.21.15.203:0/3934980039 >> 172.21.15.203:0/3934980039 conn(0x7f35380f09a0 msgr2=0x7f35380f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 -- 172.21.15.203:0/3934980039 shutdown_connections 2024-02-19T22:16:22.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.556+0000 7f353c98d700 1 -- 172.21.15.203:0/3934980039 wait complete. 2024-02-19T22:16:22.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.557+0000 7f353c98d700 1 Processor -- start 2024-02-19T22:16:22.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.557+0000 7f353c98d700 1 -- start start 2024-02-19T22:16:22.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.557+0000 7f353c98d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:22.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.557+0000 7f353c98d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35380f13e0 con 0x7f35380f51a0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f3536d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f3536d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34406/0 (socket says 172.21.15.203:34406) 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f3536d9d700 1 -- 172.21.15.203:0/3481382181 learned_addr learned my addr 172.21.15.203:0/3481382181 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f3536d9d700 1 -- 172.21.15.203:0/3481382181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528005040 con 0x7f35380f51a0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f3536d9d700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f35280089f0 tx=0x7f352800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3528009320 con 0x7f35380f51a0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35380f1580 con 0x7f35380f51a0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3528013380 con 0x7f35380f51a0 2024-02-19T22:16:22.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3528014e20 con 0x7f35380f51a0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.558+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35380f1a10 con 0x7f35380f51a0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.559+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3528014420 con 0x7f35380f51a0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.559+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3538059100 con 0x7f35380f51a0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.559+0000 7f351f7fe700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.559+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3528041040 con 0x7f35380f51a0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.560+0000 7f353659c700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:22.562 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.560+0000 7f353659c700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f352c0096a0 tx=0x7f352c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:22.565 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.564+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35280095f0 con 0x7f35380f51a0 2024-02-19T22:16:22.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.767+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3538045ad0 con 0x7f35380f51a0 2024-02-19T22:16:22.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.768+0000 7f351f7fe700 1 -- 172.21.15.203:0/3481382181 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f352801db70 con 0x7f35380f51a0 2024-02-19T22:16:22.769 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:22.769 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 msgr2=0x7f35200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f352c0096a0 tx=0x7f352c006b40).stop 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 msgr2=0x7f35380f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f35280089f0 tx=0x7f352800b040).stop 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 shutdown_connections 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 --2- 172.21.15.203:0/3481382181 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35380f51a0 0x7f35380f2f60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 >> 172.21.15.203:0/3481382181 conn(0x7f35380f09a0 msgr2=0x7f353805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 shutdown_connections 2024-02-19T22:16:22.771 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:22.770+0000 7f353c98d700 1 -- 172.21.15.203:0/3481382181 wait complete. 2024-02-19T22:16:22.772 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:22.768336+0000 mon.smithi111 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.203:0/3481382181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:24.418 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:24.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:24.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:23.185727+0000 mgr.smithi111.ftjzov (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 -- 172.21.15.203:0/1331121240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 msgr2=0x7fcff80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/1331121240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff80f6930 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcfe8009fb0 tx=0x7fcfe8009b30).stop 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 -- 172.21.15.203:0/1331121240 shutdown_connections 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/1331121240 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff80f6930 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 -- 172.21.15.203:0/1331121240 >> 172.21.15.203:0/1331121240 conn(0x7fcff80f1c70 msgr2=0x7fcff80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 -- 172.21.15.203:0/1331121240 shutdown_connections 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.239+0000 7fcffc9d5700 1 -- 172.21.15.203:0/1331121240 wait complete. 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcffc9d5700 1 Processor -- start 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcffc9d5700 1 -- start start 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcffc9d5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcffc9d5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff8106cd0 con 0x7fcff80f6510 2024-02-19T22:16:26.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcff659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcff659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41888/0 (socket says 172.21.15.203:41888) 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcff659c700 1 -- 172.21.15.203:0/2451599322 learned_addr learned my addr 172.21.15.203:0/2451599322 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcff659c700 1 -- 172.21.15.203:0/2451599322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe8005040 con 0x7fcff80f6510 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcff659c700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcfe8005010 tx=0x7fcfe800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe8009320 con 0x7fcff80f6510 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe8013380 con 0x7fcff80f6510 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.240+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff8106ed0 con 0x7fcff80f6510 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe8014e50 con 0x7fcff80f6510 2024-02-19T22:16:26.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff8107360 con 0x7fcff80f6510 2024-02-19T22:16:26.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcfe80095f0 con 0x7fcff80f6510 2024-02-19T22:16:26.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff8045ad0 con 0x7fcff80f6510 2024-02-19T22:16:26.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcfeeffd700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 0x7fcfe40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:26.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.241+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcfe8040fa0 con 0x7fcff80f6510 2024-02-19T22:16:26.245 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.242+0000 7fcff5d9b700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 0x7fcfe40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:26.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.242+0000 7fcff5d9b700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 0x7fcfe40323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcfe00096a0 tx=0x7fcfe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:26.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.246+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcfe8014420 con 0x7fcff80f6510 2024-02-19T22:16:26.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.451+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcff8108210 con 0x7fcff80f6510 2024-02-19T22:16:26.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.451+0000 7fcfeeffd700 1 -- 172.21.15.203:0/2451599322 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcfe8010230 con 0x7fcff80f6510 2024-02-19T22:16:26.454 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:26.454 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:26.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 msgr2=0x7fcfe40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 0x7fcfe40323d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fcfe00096a0 tx=0x7fcfe0006b40).stop 2024-02-19T22:16:26.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 msgr2=0x7fcff81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:26.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fcfe8005010 tx=0x7fcfe800b040).stop 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 shutdown_connections 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcfe402ff10 0x7fcfe40323d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 --2- 172.21.15.203:0/2451599322 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcff80f6510 0x7fcff81067e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 >> 172.21.15.203:0/2451599322 conn(0x7fcff80f1c70 msgr2=0x7fcff80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 shutdown_connections 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:26.453+0000 7fcffc9d5700 1 -- 172.21.15.203:0/2451599322 wait complete. 2024-02-19T22:16:26.456 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:26.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:25.186057+0000 mgr.smithi111.ftjzov (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:27.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:26.452780+0000 mon.smithi111 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.203:0/2451599322' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:28.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:28.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:28.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:27.186433+0000 mgr.smithi111.ftjzov (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/211548457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 msgr2=0x7ffbd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/211548457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd00f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7ffbc0009fb0 tx=0x7ffbc0009b30).stop 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/211548457 shutdown_connections 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/211548457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd00f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/211548457 >> 172.21.15.203:0/211548457 conn(0x7ffbd00f0b70 msgr2=0x7ffbd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/211548457 shutdown_connections 2024-02-19T22:16:30.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.163+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/211548457 wait complete. 2024-02-19T22:16:30.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd7aa7700 1 Processor -- start 2024-02-19T22:16:30.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd7aa7700 1 -- start start 2024-02-19T22:16:30.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd7aa7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:30.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd7aa7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd0106cd0 con 0x7ffbd00f5410 2024-02-19T22:16:30.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd5843700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd5843700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41896/0 (socket says 172.21.15.203:41896) 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.164+0000 7ffbd5843700 1 -- 172.21.15.203:0/3220979712 learned_addr learned my addr 172.21.15.203:0/3220979712 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbd5843700 1 -- 172.21.15.203:0/3220979712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc0005040 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbd5843700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ffbc0005010 tx=0x7ffbc000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbc0009320 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd0106ed0 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbc0013380 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbc0014e50 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.165+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd0107360 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbc00263a0 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbc67fc700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbc0040f70 con 0x7ffbd00f5410 2024-02-19T22:16:30.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd0059100 con 0x7ffbd00f5410 2024-02-19T22:16:30.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbd5042700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:30.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.166+0000 7ffbd5042700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:30.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.170+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbc0012920 con 0x7ffbd00f5410 2024-02-19T22:16:30.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.375+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbd00fafa0 con 0x7ffbd00f5410 2024-02-19T22:16:30.378 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.376+0000 7ffbc67fc700 1 -- 172.21.15.203:0/3220979712 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbc002b020 con 0x7ffbd00f5410 2024-02-19T22:16:30.378 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:30.378 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 msgr2=0x7ffbbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ffbcc0096a0 tx=0x7ffbcc006b40).stop 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 msgr2=0x7ffbd01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ffbc0005010 tx=0x7ffbc000b040).stop 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 shutdown_connections 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbbc02ff10 0x7ffbbc0323d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 --2- 172.21.15.203:0/3220979712 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbd00f5410 0x7ffbd01067e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 >> 172.21.15.203:0/3220979712 conn(0x7ffbd00f0b70 msgr2=0x7ffbd00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:30.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 shutdown_connections 2024-02-19T22:16:30.381 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:30.378+0000 7ffbd7aa7700 1 -- 172.21.15.203:0/3220979712 wait complete. 2024-02-19T22:16:30.381 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:30.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:29.186706+0000 mgr.smithi111.ftjzov (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:31.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:30.377477+0000 mon.smithi111 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.203:0/3220979712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:32.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:32.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:32.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:31.186956+0000 mgr.smithi111.ftjzov (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:34.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 -- 172.21.15.203:0/2680878782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 msgr2=0x7ff5300f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 --2- 172.21.15.203:0/2680878782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff5300f90f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7ff524009fb0 tx=0x7ff524009b30).stop 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 -- 172.21.15.203:0/2680878782 shutdown_connections 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 --2- 172.21.15.203:0/2680878782 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff5300f90f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 -- 172.21.15.203:0/2680878782 >> 172.21.15.203:0/2680878782 conn(0x7ff5300f0b70 msgr2=0x7ff5300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 -- 172.21.15.203:0/2680878782 shutdown_connections 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.080+0000 7ff536742700 1 -- 172.21.15.203:0/2680878782 wait complete. 2024-02-19T22:16:34.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff536742700 1 Processor -- start 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff536742700 1 -- start start 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff536742700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff536742700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff530104a60 con 0x7ff530060280 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff52ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff52ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41910/0 (socket says 172.21.15.203:41910) 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.081+0000 7ff52ffff700 1 -- 172.21.15.203:0/2706825259 learned_addr learned my addr 172.21.15.203:0/2706825259 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:34.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff52ffff700 1 -- 172.21.15.203:0/2706825259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524005040 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff52ffff700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff5240089f0 tx=0x7ff52400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff524009320 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff524013380 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff530104c60 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff524014e20 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.082+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5301050f0 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.083+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff524014420 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.083+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff530045ad0 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.083+0000 7ff52cff9700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 0x7ff518036870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.083+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff524040e80 con 0x7ff530060280 2024-02-19T22:16:34.086 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.083+0000 7ff52f7fe700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 0x7ff518036870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:34.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.084+0000 7ff52f7fe700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 0x7ff518036870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff5200096a0 tx=0x7ff520006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:34.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.088+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff524014790 con 0x7ff530060280 2024-02-19T22:16:34.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.291+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff530105fa0 con 0x7ff530060280 2024-02-19T22:16:34.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.292+0000 7ff52cff9700 1 -- 172.21.15.203:0/2706825259 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff524004490 con 0x7ff530060280 2024-02-19T22:16:34.294 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:34.294 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:34.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 msgr2=0x7ff518036870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 0x7ff518036870 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ff5200096a0 tx=0x7ff520006b40).stop 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 msgr2=0x7ff530104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7ff5240089f0 tx=0x7ff52400b040).stop 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 shutdown_connections 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.293+0000 7ff536742700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff5180343b0 0x7ff518036870 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.294+0000 7ff536742700 1 --2- 172.21.15.203:0/2706825259 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff530060280 0x7ff530104570 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.294+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 >> 172.21.15.203:0/2706825259 conn(0x7ff5300f0b70 msgr2=0x7ff5300f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.294+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 shutdown_connections 2024-02-19T22:16:34.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:34.294+0000 7ff536742700 1 -- 172.21.15.203:0/2706825259 wait complete. 2024-02-19T22:16:34.297 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:34.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:33.187276+0000 mgr.smithi111.ftjzov (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:35.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:35 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:34.293145+0000 mon.smithi111 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.203:0/2706825259' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:36.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:36.023 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:35.187604+0000 mgr.smithi111.ftjzov (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:37.915 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 -- 172.21.15.203:0/2602664254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 msgr2=0x7f23600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 --2- 172.21.15.203:0/2602664254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23600f6930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2354009fb0 tx=0x7f2354009b30).stop 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 -- 172.21.15.203:0/2602664254 shutdown_connections 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 --2- 172.21.15.203:0/2602664254 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23600f6930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 -- 172.21.15.203:0/2602664254 >> 172.21.15.203:0/2602664254 conn(0x7f23600f1c70 msgr2=0x7f23600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 -- 172.21.15.203:0/2602664254 shutdown_connections 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.913+0000 7f23654c6700 1 -- 172.21.15.203:0/2602664254 wait complete. 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f23654c6700 1 Processor -- start 2024-02-19T22:16:37.916 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f23654c6700 1 -- start start 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f23654c6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f23654c6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360106cd0 con 0x7f23600f6510 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f235effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.914+0000 7f235effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58908/0 (socket says 172.21.15.203:58908) 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f235effd700 1 -- 172.21.15.203:0/1450697930 learned_addr learned my addr 172.21.15.203:0/1450697930 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:37.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f235effd700 1 -- 172.21.15.203:0/1450697930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2354005040 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f235effd700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2354005010 tx=0x7f235400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2354009320 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2360106ed0 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2354013380 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2354014e50 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.915+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2360107360 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.916+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23540095f0 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.916+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2360045ad0 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.916+0000 7f23477fe700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 0x7f23480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.916+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2354040fa0 con 0x7f23600f6510 2024-02-19T22:16:37.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.916+0000 7f235e7fc700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 0x7f23480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:37.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.917+0000 7f235e7fc700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 0x7f23480323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f23500096a0 tx=0x7f2350006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:37.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:37.921+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2354014420 con 0x7f23600f6510 2024-02-19T22:16:38.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.124+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2360108210 con 0x7f23600f6510 2024-02-19T22:16:38.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.124+0000 7f23477fe700 1 -- 172.21.15.203:0/1450697930 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2354010230 con 0x7f23600f6510 2024-02-19T22:16:38.127 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:38.127 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:38.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 msgr2=0x7f23480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:38.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 0x7f23480323d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f23500096a0 tx=0x7f2350006b40).stop 2024-02-19T22:16:38.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 msgr2=0x7f23601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:38.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f2354005010 tx=0x7f235400b040).stop 2024-02-19T22:16:38.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 shutdown_connections 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f234802ff10 0x7f23480323d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 --2- 172.21.15.203:0/1450697930 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f23600f6510 0x7f23601067e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 >> 172.21.15.203:0/1450697930 conn(0x7f23600f1c70 msgr2=0x7f23600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 shutdown_connections 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:38.126+0000 7f23654c6700 1 -- 172.21.15.203:0/1450697930 wait complete. 2024-02-19T22:16:38.129 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:37.187896+0000 mgr.smithi111.ftjzov (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:38.125686+0000 mon.smithi111 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.203:0/1450697930' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:39.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:39.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:39.188254+0000 mgr.smithi111.ftjzov (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:41.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 -- 172.21.15.203:0/2498897492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 msgr2=0x7f03040f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 --2- 172.21.15.203:0/2498897492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f4be0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f02fc009fb0 tx=0x7f02fc009b30).stop 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 -- 172.21.15.203:0/2498897492 shutdown_connections 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 --2- 172.21.15.203:0/2498897492 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f4be0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 -- 172.21.15.203:0/2498897492 >> 172.21.15.203:0/2498897492 conn(0x7f03040f02e0 msgr2=0x7f03040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 -- 172.21.15.203:0/2498897492 shutdown_connections 2024-02-19T22:16:41.541 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.538+0000 7f03090bf700 1 -- 172.21.15.203:0/2498897492 wait complete. 2024-02-19T22:16:41.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.539+0000 7f03090bf700 1 Processor -- start 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.539+0000 7f03090bf700 1 -- start start 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.539+0000 7f03090bf700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.539+0000 7f03090bf700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03040f57f0 con 0x7f03040f47c0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f0302d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f0302d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58924/0 (socket says 172.21.15.203:58924) 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f0302d9d700 1 -- 172.21.15.203:0/1135156911 learned_addr learned my addr 172.21.15.203:0/1135156911 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f0302d9d700 1 -- 172.21.15.203:0/1135156911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc005040 con 0x7f03040f47c0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f0302d9d700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f02fc0089f0 tx=0x7f02fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02fc009320 con 0x7f03040f47c0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02fc013380 con 0x7f03040f47c0 2024-02-19T22:16:41.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03040f5990 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02fc01ed80 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.540+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03040f5e20 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.541+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02fc01e420 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.541+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304059100 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.541+0000 7f02fb7fe700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.541+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02fc041090 con 0x7f03040f47c0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.542+0000 7f030259c700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:41.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.542+0000 7f030259c700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 0x7f02f0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:41.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.546+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02fc0095f0 con 0x7f03040f47c0 2024-02-19T22:16:41.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.752+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03040fc090 con 0x7f03040f47c0 2024-02-19T22:16:41.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.752+0000 7f02fb7fe700 1 -- 172.21.15.203:0/1135156911 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f02fc004490 con 0x7f03040f47c0 2024-02-19T22:16:41.755 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:41.755 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 msgr2=0x7f02f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 0x7f02f0032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f02ec0096a0 tx=0x7f02ec006b40).stop 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 msgr2=0x7f03040f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f02fc0089f0 tx=0x7f02fc00b040).stop 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 shutdown_connections 2024-02-19T22:16:41.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02f002ff60 0x7f02f0032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 --2- 172.21.15.203:0/1135156911 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03040f47c0 0x7f03040f7370 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:41.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 >> 172.21.15.203:0/1135156911 conn(0x7f03040f02e0 msgr2=0x7f03040f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:41.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 shutdown_connections 2024-02-19T22:16:41.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:41.754+0000 7f03090bf700 1 -- 172.21.15.203:0/1135156911 wait complete. 2024-02-19T22:16:41.760 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:42.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:41.188458+0000 mgr.smithi111.ftjzov (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:42.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:41.753655+0000 mon.smithi111 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.203:0/1135156911' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:43.447 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:43.447 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:44.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:43.188808+0000 mgr.smithi111.ftjzov (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:45.341 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 -- 172.21.15.203:0/79086270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 msgr2=0x7fb6cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/79086270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc0f66c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fb6bc009fb0 tx=0x7fb6bc009b30).stop 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 -- 172.21.15.203:0/79086270 shutdown_connections 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/79086270 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc0f66c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 -- 172.21.15.203:0/79086270 >> 172.21.15.203:0/79086270 conn(0x7fb6cc0f1aa0 msgr2=0x7fb6cc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 -- 172.21.15.203:0/79086270 shutdown_connections 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.337+0000 7fb6d17cd700 1 -- 172.21.15.203:0/79086270 wait complete. 2024-02-19T22:16:45.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6d17cd700 1 Processor -- start 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6d17cd700 1 -- start start 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6d17cd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6d17cd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6cc106ac0 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58550/0 (socket says 172.21.15.203:58550) 2024-02-19T22:16:45.343 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.338+0000 7fb6cbfff700 1 -- 172.21.15.203:0/1697001236 learned_addr learned my addr 172.21.15.203:0/1697001236 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6cbfff700 1 -- 172.21.15.203:0/1697001236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc005040 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6cbfff700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6bc0089f0 tx=0x7fb6bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6bc009320 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6cc106cc0 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6bc013380 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6bc014e20 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.339+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6cc107150 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.340+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6bc0263a0 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.340+0000 7fb6c8ff9700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 0x7fb6b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:45.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.340+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6bc040f60 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.340+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6cc059100 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.340+0000 7fb6cb7fe700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 0x7fb6b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:45.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.341+0000 7fb6cb7fe700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 0x7fb6b40323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6c00096a0 tx=0x7fb6c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:45.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.345+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6bc012920 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.549+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6cc0fbe30 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.550+0000 7fb6c8ff9700 1 -- 172.21.15.203:0/1697001236 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6bc02b020 con 0x7fb6cc0f62a0 2024-02-19T22:16:45.553 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:45.553 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:45.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 msgr2=0x7fb6b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 0x7fb6b40323d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fb6c00096a0 tx=0x7fb6c0006b40).stop 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 msgr2=0x7fb6cc1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fb6bc0089f0 tx=0x7fb6bc00b040).stop 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 shutdown_connections 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb6b402ff10 0x7fb6b40323d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 --2- 172.21.15.203:0/1697001236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6cc0f62a0 0x7fb6cc1065d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.551+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 >> 172.21.15.203:0/1697001236 conn(0x7fb6cc0f1aa0 msgr2=0x7fb6cc0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.552+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 shutdown_connections 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:45.552+0000 7fb6d17cd700 1 -- 172.21.15.203:0/1697001236 wait complete. 2024-02-19T22:16:45.554 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:45.189150+0000 mgr.smithi111.ftjzov (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:46.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:45.551063+0000 mon.smithi111 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.203:0/1697001236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:47.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:47.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:48.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:47.189519+0000 mgr.smithi111.ftjzov (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:49.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 -- 172.21.15.203:0/2961368118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc0f5410 msgr2=0x7f3abc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:49.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 --2- 172.21.15.203:0/2961368118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc0f5410 0x7f3abc0f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f3ab0009fb0 tx=0x7f3ab0009b30).stop 2024-02-19T22:16:49.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 -- 172.21.15.203:0/2961368118 shutdown_connections 2024-02-19T22:16:49.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 --2- 172.21.15.203:0/2961368118 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc0f5410 0x7f3abc0f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:49.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 -- 172.21.15.203:0/2961368118 >> 172.21.15.203:0/2961368118 conn(0x7f3abc0f0b70 msgr2=0x7f3abc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:49.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 -- 172.21.15.203:0/2961368118 shutdown_connections 2024-02-19T22:16:49.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.198+0000 7f3ac303f700 1 -- 172.21.15.203:0/2961368118 wait complete. 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.199+0000 7f3ac303f700 1 Processor -- start 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.199+0000 7f3ac303f700 1 -- start start 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.199+0000 7f3ac303f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.199+0000 7f3ac303f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc109b30 con 0x7f3abc060200 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac0ddb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:49.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac0ddb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58564/0 (socket says 172.21.15.203:58564) 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac0ddb700 1 -- 172.21.15.203:0/3543889026 learned_addr learned my addr 172.21.15.203:0/3543889026 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac0ddb700 1 -- 172.21.15.203:0/3543889026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab0005040 con 0x7f3abc060200 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac0ddb700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3abc0f6510 tx=0x7f3ab0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab0013380 con 0x7f3abc060200 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ab0013790 con 0x7f3abc060200 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc109c70 con 0x7f3abc060200 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ab001ee70 con 0x7f3abc060200 2024-02-19T22:16:49.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.200+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc10a0a0 con 0x7f3abc060200 2024-02-19T22:16:49.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.201+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ab0025060 con 0x7f3abc060200 2024-02-19T22:16:49.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.201+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc045ad0 con 0x7f3abc060200 2024-02-19T22:16:49.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.201+0000 7f3ab97fa700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 0x7f3aac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:49.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.201+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ab0041160 con 0x7f3abc060200 2024-02-19T22:16:49.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.201+0000 7f3abbfff700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 0x7f3aac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:49.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.202+0000 7f3abbfff700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 0x7f3aac032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3aa80096a0 tx=0x7f3aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:49.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.206+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ab001c750 con 0x7f3abc060200 2024-02-19T22:16:49.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.412+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3abc0fafa0 con 0x7f3abc060200 2024-02-19T22:16:49.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.413+0000 7f3ab97fa700 1 -- 172.21.15.203:0/3543889026 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ab002d2d0 con 0x7f3abc060200 2024-02-19T22:16:49.415 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:49.415 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:49.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 msgr2=0x7f3aac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:49.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 0x7f3aac032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f3aa80096a0 tx=0x7f3aa8006b40).stop 2024-02-19T22:16:49.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 msgr2=0x7f3abc109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f3abc0f6510 tx=0x7f3ab0009b30).stop 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 shutdown_connections 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3aac02ff60 0x7f3aac032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 --2- 172.21.15.203:0/3543889026 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3abc060200 0x7f3abc109640 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 >> 172.21.15.203:0/3543889026 conn(0x7f3abc0f0b70 msgr2=0x7f3abc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 shutdown_connections 2024-02-19T22:16:49.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:49.414+0000 7f3ac303f700 1 -- 172.21.15.203:0/3543889026 wait complete. 2024-02-19T22:16:49.418 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:50.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:49.189880+0000 mgr.smithi111.ftjzov (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:50.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:49.414098+0000 mon.smithi111 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.203:0/3543889026' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:51.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:51.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:52.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:51.190154+0000 mgr.smithi111.ftjzov (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- 172.21.15.203:0/3344603175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 msgr2=0x7fb4c40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3344603175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fb4b8009fb0 tx=0x7fb4b8009b30).stop 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- 172.21.15.203:0/3344603175 shutdown_connections 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3344603175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- 172.21.15.203:0/3344603175 >> 172.21.15.203:0/3344603175 conn(0x7fb4c40f0b70 msgr2=0x7fb4c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:52.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- 172.21.15.203:0/3344603175 shutdown_connections 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- 172.21.15.203:0/3344603175 wait complete. 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 Processor -- start 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.987+0000 7fb4cade9700 1 -- start start 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4cade9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4cade9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c40fb960 con 0x7fb4c40f5410 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c8b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c8b85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58578/0 (socket says 172.21.15.203:58578) 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c8b85700 1 -- 172.21.15.203:0/3421246903 learned_addr learned my addr 172.21.15.203:0/3421246903 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c8b85700 1 -- 172.21.15.203:0/3421246903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8005040 con 0x7fb4c40f5410 2024-02-19T22:16:52.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c8b85700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb4b8005010 tx=0x7fb4b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.992 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.988+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b8009320 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c40fbb00 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4b8013380 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4b8014e50 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c40fbf90 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4b80095f0 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.989+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c4059100 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.990+0000 7fb4c17fa700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 0x7fb4b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.990+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4b8040e70 con 0x7fb4c40f5410 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.990+0000 7fb4c3fff700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 0x7fb4b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:52.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.990+0000 7fb4c3fff700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 0x7fb4b40323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb4b00096a0 tx=0x7fb4b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:52.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:52.994+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4b8014420 con 0x7fb4c40f5410 2024-02-19T22:16:53.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.201+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb4c4045ad0 con 0x7fb4c40f5410 2024-02-19T22:16:53.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.201+0000 7fb4c17fa700 1 -- 172.21.15.203:0/3421246903 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb4b8010230 con 0x7fb4c40f5410 2024-02-19T22:16:53.203 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:53.203 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 msgr2=0x7fb4b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 0x7fb4b40323d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fb4b00096a0 tx=0x7fb4b0006b40).stop 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 msgr2=0x7fb4c40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fb4b8005010 tx=0x7fb4b800b040).stop 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 shutdown_connections 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb4b402ff10 0x7fb4b40323d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 --2- 172.21.15.203:0/3421246903 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb4c40f5410 0x7fb4c40fd4e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:53.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 >> 172.21.15.203:0/3421246903 conn(0x7fb4c40f0b70 msgr2=0x7fb4c405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:53.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 shutdown_connections 2024-02-19T22:16:53.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:53.203+0000 7fb4cade9700 1 -- 172.21.15.203:0/3421246903 wait complete. 2024-02-19T22:16:53.206 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:54.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:53.190509+0000 mgr.smithi111.ftjzov (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:54.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:53.202618+0000 mon.smithi111 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.203:0/3421246903' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:54.928 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:54.929 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:16:56.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:55.190805+0000 mgr.smithi111.ftjzov (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:57.026 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- 172.21.15.203:0/124803888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 msgr2=0x7f88200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 --2- 172.21.15.203:0/124803888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f55c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8810009fb0 tx=0x7f8810009b30).stop 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- 172.21.15.203:0/124803888 shutdown_connections 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 --2- 172.21.15.203:0/124803888 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f55c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- 172.21.15.203:0/124803888 >> 172.21.15.203:0/124803888 conn(0x7f88200f09a0 msgr2=0x7f88200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- 172.21.15.203:0/124803888 shutdown_connections 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- 172.21.15.203:0/124803888 wait complete. 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 Processor -- start 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.024+0000 7f882488e700 1 -- start start 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f882488e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f882488e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200f13e0 con 0x7f88200f51a0 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f881ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f881ed9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50348/0 (socket says 172.21.15.203:50348) 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f881ed9d700 1 -- 172.21.15.203:0/3156373428 learned_addr learned my addr 172.21.15.203:0/3156373428 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f881ed9d700 1 -- 172.21.15.203:0/3156373428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8810005040 con 0x7f88200f51a0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f881ed9d700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f88100089f0 tx=0x7f881000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8810009320 con 0x7f88200f51a0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88200f1580 con 0x7f88200f51a0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8810013380 con 0x7f88200f51a0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8810014e20 con 0x7f88200f51a0 2024-02-19T22:16:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.025+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88200f1a10 con 0x7f88200f51a0 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.026+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88100263a0 con 0x7f88200f51a0 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.026+0000 7f88077fe700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 0x7f88080323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.026+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8820059100 con 0x7f88200f51a0 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.026+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8810040f00 con 0x7f88200f51a0 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.027+0000 7f881e59c700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 0x7f88080323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:16:57.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.027+0000 7f881e59c700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 0x7f88080323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f88140096a0 tx=0x7f8814006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:16:57.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.031+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8810012330 con 0x7f88200f51a0 2024-02-19T22:16:57.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.239+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8820045ad0 con 0x7f88200f51a0 2024-02-19T22:16:57.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.240+0000 7f88077fe700 1 -- 172.21.15.203:0/3156373428 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f881001db70 con 0x7f88200f51a0 2024-02-19T22:16:57.241 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:16:57.241 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:16:57.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 msgr2=0x7f88080323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:57.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 0x7f88080323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f88140096a0 tx=0x7f8814006b40).stop 2024-02-19T22:16:57.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 msgr2=0x7f88200f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f88100089f0 tx=0x7f881000b040).stop 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 shutdown_connections 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f880802ff10 0x7f88080323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 --2- 172.21.15.203:0/3156373428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f88200f51a0 0x7f88200f2f60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 >> 172.21.15.203:0/3156373428 conn(0x7f88200f09a0 msgr2=0x7f882005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 shutdown_connections 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:16:57.241+0000 7f882488e700 1 -- 172.21.15.203:0/3156373428 wait complete. 2024-02-19T22:16:57.243 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:16:58.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:57.191094+0000 mgr.smithi111.ftjzov (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:16:58.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:16:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:16:57.239976+0000 mon.smithi111 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.203:0/3156373428' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:16:58.919 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:16:58.919 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:00.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:16:59.191433+0000 mgr.smithi111.ftjzov (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:00.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 -- 172.21.15.203:0/3125727442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 msgr2=0x7fb93c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 --2- 172.21.15.203:0/3125727442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c0f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fb930009fb0 tx=0x7fb930009b30).stop 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 -- 172.21.15.203:0/3125727442 shutdown_connections 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 --2- 172.21.15.203:0/3125727442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c0f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 -- 172.21.15.203:0/3125727442 >> 172.21.15.203:0/3125727442 conn(0x7fb93c0f1c70 msgr2=0x7fb93c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 -- 172.21.15.203:0/3125727442 shutdown_connections 2024-02-19T22:17:00.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.760+0000 7fb94303b700 1 -- 172.21.15.203:0/3125727442 wait complete. 2024-02-19T22:17:00.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.761+0000 7fb94303b700 1 Processor -- start 2024-02-19T22:17:00.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.761+0000 7fb94303b700 1 -- start start 2024-02-19T22:17:00.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.761+0000 7fb94303b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:00.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.761+0000 7fb94303b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93c108ed0 con 0x7fb93c0f6510 2024-02-19T22:17:00.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb940dd7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb940dd7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50358/0 (socket says 172.21.15.203:50358) 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb940dd7700 1 -- 172.21.15.203:0/3520694369 learned_addr learned my addr 172.21.15.203:0/3520694369 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb940dd7700 1 -- 172.21.15.203:0/3520694369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930005040 con 0x7fb93c0f6510 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb940dd7700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb930005010 tx=0x7fb93000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb930009320 con 0x7fb93c0f6510 2024-02-19T22:17:00.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93c1090d0 con 0x7fb93c0f6510 2024-02-19T22:17:00.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93c109560 con 0x7fb93c0f6510 2024-02-19T22:17:00.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb930013380 con 0x7fb93c0f6510 2024-02-19T22:17:00.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.762+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb930014df0 con 0x7fb93c0f6510 2024-02-19T22:17:00.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.763+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb93001e420 con 0x7fb93c0f6510 2024-02-19T22:17:00.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.763+0000 7fb9397fa700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 0x7fb92c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:00.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.763+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93c045ad0 con 0x7fb93c0f6510 2024-02-19T22:17:00.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.763+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb930040ed0 con 0x7fb93c0f6510 2024-02-19T22:17:00.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.764+0000 7fb93bfff700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 0x7fb92c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:00.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.764+0000 7fb93bfff700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 0x7fb92c0323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9280096a0 tx=0x7fb928006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:00.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.768+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9300263a0 con 0x7fb93c0f6510 2024-02-19T22:17:00.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.972+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb93c10a410 con 0x7fb93c0f6510 2024-02-19T22:17:00.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.973+0000 7fb9397fa700 1 -- 172.21.15.203:0/3520694369 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb93001c970 con 0x7fb93c0f6510 2024-02-19T22:17:00.974 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:00.974 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:00.975 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 msgr2=0x7fb92c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 0x7fb92c0323d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb9280096a0 tx=0x7fb928006b40).stop 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 msgr2=0x7fb93c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fb930005010 tx=0x7fb93000b040).stop 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 shutdown_connections 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb92c02ff10 0x7fb92c0323d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 --2- 172.21.15.203:0/3520694369 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb93c0f6510 0x7fb93c1089e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 >> 172.21.15.203:0/3520694369 conn(0x7fb93c0f1c70 msgr2=0x7fb93c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 shutdown_connections 2024-02-19T22:17:00.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:00.974+0000 7fb94303b700 1 -- 172.21.15.203:0/3520694369 wait complete. 2024-02-19T22:17:00.977 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:01.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:00.972775+0000 mon.smithi111 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.203:0/3520694369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:02.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:01.191756+0000 mgr.smithi111.ftjzov (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:02.633 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:02.633 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:04.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:03.192109+0000 mgr.smithi111.ftjzov (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- 172.21.15.203:0/1524471345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 msgr2=0x7feaac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 --2- 172.21.15.203:0/1524471345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac0f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fea9c009fb0 tx=0x7fea9c009b30).stop 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- 172.21.15.203:0/1524471345 shutdown_connections 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 --2- 172.21.15.203:0/1524471345 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac0f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- 172.21.15.203:0/1524471345 >> 172.21.15.203:0/1524471345 conn(0x7feaac0f0b70 msgr2=0x7feaac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- 172.21.15.203:0/1524471345 shutdown_connections 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- 172.21.15.203:0/1524471345 wait complete. 2024-02-19T22:17:04.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 Processor -- start 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- start start 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.479+0000 7feab370b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac105bd0 con 0x7feaac0f5410 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab14a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab14a7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50370/0 (socket says 172.21.15.203:50370) 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab14a7700 1 -- 172.21.15.203:0/557846319 learned_addr learned my addr 172.21.15.203:0/557846319 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab14a7700 1 -- 172.21.15.203:0/557846319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9c005040 con 0x7feaac0f5410 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab14a7700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fea9c0089f0 tx=0x7fea9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea9c009320 con 0x7feaac0f5410 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac105dd0 con 0x7feaac0f5410 2024-02-19T22:17:04.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea9c013380 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea9c01ed80 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.480+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac106260 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.481+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac045ad0 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.481+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea9c01e420 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.481+0000 7feaa1ffb700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 0x7fea98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.481+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea9c040ff0 con 0x7feaac0f5410 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.482+0000 7feab0ca6700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 0x7fea98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.482+0000 7feab0ca6700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 0x7fea98032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7feaa80096a0 tx=0x7feaa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:04.527 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.486+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea9c0095f0 con 0x7feaac0f5410 2024-02-19T22:17:04.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.691+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaac1070b0 con 0x7feaac0f5410 2024-02-19T22:17:04.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.691+0000 7feaa1ffb700 1 -- 172.21.15.203:0/557846319 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea9c01cb10 con 0x7feaac0f5410 2024-02-19T22:17:04.693 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:04.694 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 msgr2=0x7fea98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 0x7fea98032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7feaa80096a0 tx=0x7feaa8006b40).stop 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 msgr2=0x7feaac1056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7fea9c0089f0 tx=0x7fea9c00b040).stop 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 shutdown_connections 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fea9802ff60 0x7fea98032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:04.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 --2- 172.21.15.203:0/557846319 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feaac0f5410 0x7feaac1056e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 >> 172.21.15.203:0/557846319 conn(0x7feaac0f0b70 msgr2=0x7feaac0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 shutdown_connections 2024-02-19T22:17:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:04.693+0000 7feab370b700 1 -- 172.21.15.203:0/557846319 wait complete. 2024-02-19T22:17:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:05.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:04.692611+0000 mon.smithi111 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.203:0/557846319' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:06.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:06.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:06.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:05.192429+0000 mgr.smithi111.ftjzov (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- 172.21.15.203:0/436757220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 msgr2=0x7f089c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/436757220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c0f55c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f0898009fb0 tx=0x7f0898009b30).stop 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- 172.21.15.203:0/436757220 shutdown_connections 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/436757220 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c0f55c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- 172.21.15.203:0/436757220 >> 172.21.15.203:0/436757220 conn(0x7f089c0f09a0 msgr2=0x7f089c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- 172.21.15.203:0/436757220 shutdown_connections 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- 172.21.15.203:0/436757220 wait complete. 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 Processor -- start 2024-02-19T22:17:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.185+0000 7f08a2ca4700 1 -- start start 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a2ca4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a2ca4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c105990 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a1ca2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a1ca2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54146/0 (socket says 172.21.15.203:54146) 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a1ca2700 1 -- 172.21.15.203:0/2397102470 learned_addr learned my addr 172.21.15.203:0/2397102470 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a1ca2700 1 -- 172.21.15.203:0/2397102470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0898005040 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a1ca2700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f08980089f0 tx=0x7f089800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0898009320 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c105b90 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0898013380 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0898014e20 con 0x7f089c0f51a0 2024-02-19T22:17:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.186+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c106020 con 0x7f089c0f51a0 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.187+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08980095f0 con 0x7f089c0f51a0 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.187+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c045ad0 con 0x7f089c0f51a0 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.187+0000 7f08927fc700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 0x7f08880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.187+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0898040f50 con 0x7f089c0f51a0 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.188+0000 7f08a14a1700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 0x7f08880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:08.191 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.188+0000 7f08a14a1700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 0x7f08880323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f088c0096a0 tx=0x7f088c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:08.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.192+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0898014420 con 0x7f089c0f51a0 2024-02-19T22:17:08.397 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.395+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f089c106ed0 con 0x7f089c0f51a0 2024-02-19T22:17:08.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.396+0000 7f08927fc700 1 -- 172.21.15.203:0/2397102470 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f089801d980 con 0x7f089c0f51a0 2024-02-19T22:17:08.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:08.398 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 msgr2=0x7f08880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 0x7f08880323d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f088c0096a0 tx=0x7f088c006b40).stop 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 msgr2=0x7f089c1054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f08980089f0 tx=0x7f089800b040).stop 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 shutdown_connections 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088802ff10 0x7f08880323d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 --2- 172.21.15.203:0/2397102470 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f51a0 0x7f089c1054a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.397+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 >> 172.21.15.203:0/2397102470 conn(0x7f089c0f09a0 msgr2=0x7f089c0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:08.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.398+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 shutdown_connections 2024-02-19T22:17:08.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:08.398+0000 7f08a2ca4700 1 -- 172.21.15.203:0/2397102470 wait complete. 2024-02-19T22:17:08.401 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:08.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:07.192768+0000 mgr.smithi111.ftjzov (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:09.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:09 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:08.397330+0000 mon.smithi111 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.203:0/2397102470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:10.008 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:10.008 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:09.193092+0000 mgr.smithi111.ftjzov (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.875+0000 7fdd51514700 1 -- 172.21.15.203:0/782289390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 msgr2=0x7fdd4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.875+0000 7fdd51514700 1 --2- 172.21.15.203:0/782289390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c0f5830 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fdd34009fb0 tx=0x7fdd34009b30).stop 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- 172.21.15.203:0/782289390 shutdown_connections 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 --2- 172.21.15.203:0/782289390 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c0f5830 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- 172.21.15.203:0/782289390 >> 172.21.15.203:0/782289390 conn(0x7fdd4c0f0b70 msgr2=0x7fdd4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- 172.21.15.203:0/782289390 shutdown_connections 2024-02-19T22:17:11.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- 172.21.15.203:0/782289390 wait complete. 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 Processor -- start 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- start start 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.876+0000 7fdd51514700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c106ce0 con 0x7fdd4c0f5410 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54154/0 (socket says 172.21.15.203:54154) 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd4affd700 1 -- 172.21.15.203:0/4792105 learned_addr learned my addr 172.21.15.203:0/4792105 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd4affd700 1 -- 172.21.15.203:0/4792105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34005040 con 0x7fdd4c0f5410 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd4affd700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fdd34005010 tx=0x7fdd3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.880 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34009320 con 0x7fdd4c0f5410 2024-02-19T22:17:11.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4c106ee0 con 0x7fdd4c0f5410 2024-02-19T22:17:11.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdd34013380 con 0x7fdd4c0f5410 2024-02-19T22:17:11.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34014e50 con 0x7fdd4c0f5410 2024-02-19T22:17:11.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.877+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c107370 con 0x7fdd4c0f5410 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.878+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdd340095f0 con 0x7fdd4c0f5410 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.878+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c045ad0 con 0x7fdd4c0f5410 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.878+0000 7fdd437fe700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 0x7fdd380323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.878+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdd34040e90 con 0x7fdd4c0f5410 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.879+0000 7fdd4a7fc700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 0x7fdd380323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:11.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.879+0000 7fdd4a7fc700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 0x7fdd380323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdd3c0096a0 tx=0x7fdd3c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:11.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:11.883+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdd34014420 con 0x7fdd4c0f5410 2024-02-19T22:17:12.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.088+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd4c108220 con 0x7fdd4c0f5410 2024-02-19T22:17:12.091 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.089+0000 7fdd437fe700 1 -- 172.21.15.203:0/4792105 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdd34010230 con 0x7fdd4c0f5410 2024-02-19T22:17:12.091 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:12.091 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:12.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 msgr2=0x7fdd380323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:12.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 0x7fdd380323d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fdd3c0096a0 tx=0x7fdd3c006b40).stop 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 msgr2=0x7fdd4c1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fdd34005010 tx=0x7fdd3400b040).stop 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 shutdown_connections 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdd3802ff10 0x7fdd380323d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 --2- 172.21.15.203:0/4792105 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdd4c0f5410 0x7fdd4c1067f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 >> 172.21.15.203:0/4792105 conn(0x7fdd4c0f0b70 msgr2=0x7fdd4c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 shutdown_connections 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:12.090+0000 7fdd51514700 1 -- 172.21.15.203:0/4792105 wait complete. 2024-02-19T22:17:12.093 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:11.193381+0000 mgr.smithi111.ftjzov (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:12.089996+0000 mon.smithi111 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.203:0/4792105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:13.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:13.735 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:13.193695+0000 mgr.smithi111.ftjzov (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 -- 172.21.15.203:0/448338312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 msgr2=0x7fbbd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 --2- 172.21.15.203:0/448338312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd80f5830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fbbc8009fb0 tx=0x7fbbc8009b30).stop 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 -- 172.21.15.203:0/448338312 shutdown_connections 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 --2- 172.21.15.203:0/448338312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd80f5830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 -- 172.21.15.203:0/448338312 >> 172.21.15.203:0/448338312 conn(0x7fbbd80f0b70 msgr2=0x7fbbd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 -- 172.21.15.203:0/448338312 shutdown_connections 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.439+0000 7fbbde82a700 1 -- 172.21.15.203:0/448338312 wait complete. 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbde82a700 1 Processor -- start 2024-02-19T22:17:15.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbde82a700 1 -- start start 2024-02-19T22:17:15.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbde82a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbde82a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd8105c00 con 0x7fbbd80f5410 2024-02-19T22:17:15.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbd7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbd7fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39482/0 (socket says 172.21.15.203:39482) 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.440+0000 7fbbd7fff700 1 -- 172.21.15.203:0/1912740996 learned_addr learned my addr 172.21.15.203:0/1912740996 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbd7fff700 1 -- 172.21.15.203:0/1912740996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc8005040 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbd7fff700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbbc8005010 tx=0x7fbbc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbc8009320 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd8105e00 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbc8013380 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbc8014e50 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.441+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd8106290 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.442+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbc8014420 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.442+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd8045ad0 con 0x7fbbd80f5410 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.442+0000 7fbbd4ff9700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 0x7fbbc00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:15.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.442+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbc8042d30 con 0x7fbbd80f5410 2024-02-19T22:17:15.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.442+0000 7fbbd77fe700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 0x7fbbc00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:15.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.443+0000 7fbbd77fe700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 0x7fbbc00323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbbcc0096a0 tx=0x7fbbcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:15.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.447+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbc8048020 con 0x7fbbd80f5410 2024-02-19T22:17:15.656 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.654+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbd8107140 con 0x7fbbd80f5410 2024-02-19T22:17:15.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.655+0000 7fbbd4ff9700 1 -- 172.21.15.203:0/1912740996 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbc80095f0 con 0x7fbbd80f5410 2024-02-19T22:17:15.657 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:15.657 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 msgr2=0x7fbbc00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 0x7fbbc00323d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fbbcc0096a0 tx=0x7fbbcc006b40).stop 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 msgr2=0x7fbbd8105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fbbc8005010 tx=0x7fbbc800b040).stop 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 shutdown_connections 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc002ff10 0x7fbbc00323d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 --2- 172.21.15.203:0/1912740996 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbd80f5410 0x7fbbd8105710 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 >> 172.21.15.203:0/1912740996 conn(0x7fbbd80f0b70 msgr2=0x7fbbd80f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 shutdown_connections 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:15.656+0000 7fbbde82a700 1 -- 172.21.15.203:0/1912740996 wait complete. 2024-02-19T22:17:15.659 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:15.194031+0000 mgr.smithi111.ftjzov (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:15.655956+0000 mon.smithi111 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.203:0/1912740996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:17.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:17.297 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:17.194298+0000 mgr.smithi111.ftjzov (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:17.299025+0000 mon.smithi111 (mon.0) 177 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:17:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:17.301494+0000 mon.smithi111 (mon.0) 178 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.154+0000 7faf03974700 1 -- 172.21.15.203:0/3675727201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 msgr2=0x7faefc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.154+0000 7faf03974700 1 --2- 172.21.15.203:0/3675727201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc0f6930 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7faeec009fb0 tx=0x7faeec009b30).stop 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- 172.21.15.203:0/3675727201 shutdown_connections 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 --2- 172.21.15.203:0/3675727201 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc0f6930 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- 172.21.15.203:0/3675727201 >> 172.21.15.203:0/3675727201 conn(0x7faefc0f1c70 msgr2=0x7faefc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- 172.21.15.203:0/3675727201 shutdown_connections 2024-02-19T22:17:19.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- 172.21.15.203:0/3675727201 wait complete. 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 Processor -- start 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- start start 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.155+0000 7faf03974700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faefc106cd0 con 0x7faefc0f6510 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf01710700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf01710700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39498/0 (socket says 172.21.15.203:39498) 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf01710700 1 -- 172.21.15.203:0/1601672106 learned_addr learned my addr 172.21.15.203:0/1601672106 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf01710700 1 -- 172.21.15.203:0/1601672106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec005040 con 0x7faefc0f6510 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf01710700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faeec0089f0 tx=0x7faeec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faeec009320 con 0x7faefc0f6510 2024-02-19T22:17:19.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faefc106ed0 con 0x7faefc0f6510 2024-02-19T22:17:19.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faeec013380 con 0x7faefc0f6510 2024-02-19T22:17:19.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faeec01ed80 con 0x7faefc0f6510 2024-02-19T22:17:19.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.156+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faefc107360 con 0x7faefc0f6510 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.157+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faeec01e420 con 0x7faefc0f6510 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.157+0000 7faef1ffb700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 0x7faee8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.157+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faefc045ad0 con 0x7faefc0f6510 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.157+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faeec041060 con 0x7faefc0f6510 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.158+0000 7faf00f0f700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 0x7faee8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:19.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.158+0000 7faf00f0f700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 0x7faee8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7faef80096a0 tx=0x7faef8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:19.164 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.162+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faeec01cb10 con 0x7faefc0f6510 2024-02-19T22:17:19.369 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.367+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faefc1081b0 con 0x7faefc0f6510 2024-02-19T22:17:19.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.368+0000 7faef1ffb700 1 -- 172.21.15.203:0/1601672106 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faeec01cb10 con 0x7faefc0f6510 2024-02-19T22:17:19.370 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:19.370 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 msgr2=0x7faee8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 0x7faee8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7faef80096a0 tx=0x7faef8006b40).stop 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 msgr2=0x7faefc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7faeec0089f0 tx=0x7faeec00b040).stop 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 shutdown_connections 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faee802ff60 0x7faee8032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 --2- 172.21.15.203:0/1601672106 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faefc0f6510 0x7faefc1067e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 >> 172.21.15.203:0/1601672106 conn(0x7faefc0f1c70 msgr2=0x7faefc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:19.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 shutdown_connections 2024-02-19T22:17:19.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:19.370+0000 7faf03974700 1 -- 172.21.15.203:0/1601672106 wait complete. 2024-02-19T22:17:19.373 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:19.194624+0000 mgr.smithi111.ftjzov (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:19.369249+0000 mon.smithi111 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.203:0/1601672106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:21.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:21.064 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:22.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:21.194851+0000 mgr.smithi111.ftjzov (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:22.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 -- 172.21.15.203:0/4192143995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 msgr2=0x7f4be80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:22.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 --2- 172.21.15.203:0/4192143995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be80f6930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f4bdc009fb0 tx=0x7f4bdc009b30).stop 2024-02-19T22:17:22.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 -- 172.21.15.203:0/4192143995 shutdown_connections 2024-02-19T22:17:22.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 --2- 172.21.15.203:0/4192143995 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be80f6930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:22.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 -- 172.21.15.203:0/4192143995 >> 172.21.15.203:0/4192143995 conn(0x7f4be80f1c70 msgr2=0x7f4be80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.873+0000 7f4beccc4700 1 -- 172.21.15.203:0/4192143995 shutdown_connections 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.874+0000 7f4beccc4700 1 -- 172.21.15.203:0/4192143995 wait complete. 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.874+0000 7f4beccc4700 1 Processor -- start 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.874+0000 7f4beccc4700 1 -- start start 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.874+0000 7f4beccc4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.874+0000 7f4beccc4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be8106cd0 con 0x7f4be80f6510 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4be659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4be659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39514/0 (socket says 172.21.15.203:39514) 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4be659c700 1 -- 172.21.15.203:0/2163648780 learned_addr learned my addr 172.21.15.203:0/2163648780 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4be659c700 1 -- 172.21.15.203:0/2163648780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc005040 con 0x7f4be80f6510 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4be659c700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f4bdc005010 tx=0x7f4bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc009320 con 0x7f4be80f6510 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be8106ed0 con 0x7f4be80f6510 2024-02-19T22:17:22.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4bdc013380 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4bdc014e50 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.875+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be8107360 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.876+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4bdc014420 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.876+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be8045ad0 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.876+0000 7f4bd6ffd700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.876+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4bdc040e30 con 0x7f4be80f6510 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.876+0000 7f4be5d9b700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.877+0000 7f4be5d9b700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 0x7f4bd00323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4bd80096a0 tx=0x7f4bd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:22.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:22.881+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4bdc0095f0 con 0x7f4be80f6510 2024-02-19T22:17:23.089 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.087+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4be8108210 con 0x7f4be80f6510 2024-02-19T22:17:23.090 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.088+0000 7f4bd6ffd700 1 -- 172.21.15.203:0/2163648780 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4bdc01db70 con 0x7f4be80f6510 2024-02-19T22:17:23.090 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:23.090 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 msgr2=0x7f4bd00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 0x7f4bd00323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f4bd80096a0 tx=0x7f4bd8006b40).stop 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 msgr2=0x7f4be81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f4bdc005010 tx=0x7f4bdc00b040).stop 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 shutdown_connections 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4bd002ff10 0x7f4bd00323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 --2- 172.21.15.203:0/2163648780 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4be80f6510 0x7f4be81067e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:23.092 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 >> 172.21.15.203:0/2163648780 conn(0x7f4be80f1c70 msgr2=0x7f4be80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:23.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 shutdown_connections 2024-02-19T22:17:23.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:23.090+0000 7f4beccc4700 1 -- 172.21.15.203:0/2163648780 wait complete. 2024-02-19T22:17:23.093 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:23.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:23.089228+0000 mon.smithi111 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.203:0/2163648780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:24.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:24.685 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:23.195094+0000 mgr.smithi111.ftjzov (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:26.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 -- 172.21.15.203:0/2832983779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 msgr2=0x7f9af40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2832983779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af40f6930 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9ae4009fb0 tx=0x7f9ae4009b30).stop 2024-02-19T22:17:26.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 -- 172.21.15.203:0/2832983779 shutdown_connections 2024-02-19T22:17:26.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2832983779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af40f6930 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 -- 172.21.15.203:0/2832983779 >> 172.21.15.203:0/2832983779 conn(0x7f9af40f1c70 msgr2=0x7f9af40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:26.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 -- 172.21.15.203:0/2832983779 shutdown_connections 2024-02-19T22:17:26.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.510+0000 7f9af88e4700 1 -- 172.21.15.203:0/2832983779 wait complete. 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af88e4700 1 Processor -- start 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af88e4700 1 -- start start 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af88e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af88e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af4106cd0 con 0x7f9af40f6510 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55806/0 (socket says 172.21.15.203:55806) 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.511+0000 7f9af259c700 1 -- 172.21.15.203:0/2323956387 learned_addr learned my addr 172.21.15.203:0/2323956387 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9af259c700 1 -- 172.21.15.203:0/2323956387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae4005040 con 0x7f9af40f6510 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9af259c700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f9ae4005010 tx=0x7f9ae400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:26.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ae4009320 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ae4013380 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af4106ed0 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ae4014e50 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.512+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af4107360 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ae40263a0 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af4045ad0 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9ae2ffd700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 0x7f9adc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ae4040ee0 con 0x7f9af40f6510 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9af1d9b700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 0x7f9adc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:26.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.513+0000 7f9af1d9b700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 0x7f9adc0323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f9ae80096a0 tx=0x7f9ae8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:26.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.518+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ae401dd70 con 0x7f9af40f6510 2024-02-19T22:17:26.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:25.195423+0000 mgr.smithi111.ftjzov (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:26.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.724+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9af4108210 con 0x7f9af40f6510 2024-02-19T22:17:26.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.725+0000 7f9ae2ffd700 1 -- 172.21.15.203:0/2323956387 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ae401d980 con 0x7f9af40f6510 2024-02-19T22:17:26.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:26.727 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 msgr2=0x7f9adc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 0x7f9adc0323d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f9ae80096a0 tx=0x7f9ae8006b40).stop 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 msgr2=0x7f9af41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f9ae4005010 tx=0x7f9ae400b040).stop 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 shutdown_connections 2024-02-19T22:17:26.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9adc02ff10 0x7f9adc0323d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 --2- 172.21.15.203:0/2323956387 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9af40f6510 0x7f9af41067e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:26.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 >> 172.21.15.203:0/2323956387 conn(0x7f9af40f1c70 msgr2=0x7f9af40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:26.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 shutdown_connections 2024-02-19T22:17:26.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:26.726+0000 7f9af88e4700 1 -- 172.21.15.203:0/2323956387 wait complete. 2024-02-19T22:17:26.730 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:27.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:26.726005+0000 mon.smithi111 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.203:0/2323956387' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:28.316 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:28.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:27.195749+0000 mgr.smithi111.ftjzov (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:30.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 -- 172.21.15.203:0/2338940582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 msgr2=0x7f16f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 --2- 172.21.15.203:0/2338940582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f40f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f16e4009fb0 tx=0x7f16e4009b30).stop 2024-02-19T22:17:30.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 -- 172.21.15.203:0/2338940582 shutdown_connections 2024-02-19T22:17:30.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 --2- 172.21.15.203:0/2338940582 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f40f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 -- 172.21.15.203:0/2338940582 >> 172.21.15.203:0/2338940582 conn(0x7f16f40f0b70 msgr2=0x7f16f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 -- 172.21.15.203:0/2338940582 shutdown_connections 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.096+0000 7f16fbf35700 1 -- 172.21.15.203:0/2338940582 wait complete. 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16fbf35700 1 Processor -- start 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16fbf35700 1 -- start start 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16fbf35700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16fbf35700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f4060220 con 0x7f16f40f5410 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16f9cd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16f9cd1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55824/0 (socket says 172.21.15.203:55824) 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16f9cd1700 1 -- 172.21.15.203:0/3175428298 learned_addr learned my addr 172.21.15.203:0/3175428298 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.097+0000 7f16f9cd1700 1 -- 172.21.15.203:0/3175428298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4005040 con 0x7f16f40f5410 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16f9cd1700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f16e4005010 tx=0x7f16e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e4009320 con 0x7f16f40f5410 2024-02-19T22:17:30.118 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f4060420 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16e4013380 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16e4014e50 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.098+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f4109bb0 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.099+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f4059100 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.099+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f16e40095f0 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.099+0000 7f16ea7fc700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 0x7f16e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.099+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f16e4040e90 con 0x7f16f40f5410 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.099+0000 7f16f94d0700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 0x7f16e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.100+0000 7f16f94d0700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 0x7f16e00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f16f00096a0 tx=0x7f16f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:30.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.104+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f16e4014420 con 0x7f16f40f5410 2024-02-19T22:17:30.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.309+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16f40fafa0 con 0x7f16f40f5410 2024-02-19T22:17:30.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.310+0000 7f16ea7fc700 1 -- 172.21.15.203:0/3175428298 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f16e4010230 con 0x7f16f40f5410 2024-02-19T22:17:30.312 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:30.313 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 msgr2=0x7f16e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 0x7f16e00323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f16f00096a0 tx=0x7f16f0006b40).stop 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 msgr2=0x7f16f4109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f16e4005010 tx=0x7f16e400b040).stop 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 shutdown_connections 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f16e002ff10 0x7f16e00323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 --2- 172.21.15.203:0/3175428298 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f16f40f5410 0x7f16f4109500 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 >> 172.21.15.203:0/3175428298 conn(0x7f16f40f0b70 msgr2=0x7f16f40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 shutdown_connections 2024-02-19T22:17:30.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:30.312+0000 7f16fbf35700 1 -- 172.21.15.203:0/3175428298 wait complete. 2024-02-19T22:17:30.315 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:29.196129+0000 mgr.smithi111.ftjzov (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:31.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:30.311467+0000 mon.smithi111 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.203:0/3175428298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:31.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:31.921 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:32.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:31.196504+0000 mgr.smithi111.ftjzov (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:33.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.732+0000 7f5f44991700 1 -- 172.21.15.203:0/2921940140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 msgr2=0x7f5f400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:33.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.732+0000 7f5f44991700 1 --2- 172.21.15.203:0/2921940140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f400f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f5f34009fb0 tx=0x7f5f34009b30).stop 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- 172.21.15.203:0/2921940140 shutdown_connections 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 --2- 172.21.15.203:0/2921940140 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f400f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- 172.21.15.203:0/2921940140 >> 172.21.15.203:0/2921940140 conn(0x7f5f400f0b70 msgr2=0x7f5f400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- 172.21.15.203:0/2921940140 shutdown_connections 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- 172.21.15.203:0/2921940140 wait complete. 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 Processor -- start 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- start start 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.733+0000 7f5f44991700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f40106d00 con 0x7f5f400f5410 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f3e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f3e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55836/0 (socket says 172.21.15.203:55836) 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f3e59c700 1 -- 172.21.15.203:0/3023849935 learned_addr learned my addr 172.21.15.203:0/3023849935 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f3e59c700 1 -- 172.21.15.203:0/3023849935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34005040 con 0x7f5f400f5410 2024-02-19T22:17:33.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f3e59c700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5f34005010 tx=0x7f5f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:33.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f34009320 con 0x7f5f400f5410 2024-02-19T22:17:33.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40106f00 con 0x7f5f400f5410 2024-02-19T22:17:33.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f34013380 con 0x7f5f400f5410 2024-02-19T22:17:33.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f34014e50 con 0x7f5f400f5410 2024-02-19T22:17:33.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.734+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f40107390 con 0x7f5f400f5410 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.735+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f340095f0 con 0x7f5f400f5410 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.735+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f40045ad0 con 0x7f5f400f5410 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.735+0000 7f5f2effd700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 0x7f5f280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.735+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f34040e70 con 0x7f5f400f5410 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.736+0000 7f5f3dd9b700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 0x7f5f280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:33.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.736+0000 7f5f3dd9b700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 0x7f5f280323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5f300096a0 tx=0x7f5f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:33.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.741+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f34014420 con 0x7f5f400f5410 2024-02-19T22:17:33.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.948+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f400fafa0 con 0x7f5f400f5410 2024-02-19T22:17:33.950 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.948+0000 7f5f2effd700 1 -- 172.21.15.203:0/3023849935 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f34010230 con 0x7f5f400f5410 2024-02-19T22:17:33.951 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:33.951 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:33.952 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 msgr2=0x7f5f280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 0x7f5f280323d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f5f300096a0 tx=0x7f5f30006b40).stop 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 msgr2=0x7f5f40106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f5f34005010 tx=0x7f5f3400b040).stop 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 shutdown_connections 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f5f2802ff10 0x7f5f280323d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 --2- 172.21.15.203:0/3023849935 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f5f400f5410 0x7f5f40106810 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 >> 172.21.15.203:0/3023849935 conn(0x7f5f400f0b70 msgr2=0x7f5f4005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 shutdown_connections 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:33.950+0000 7f5f44991700 1 -- 172.21.15.203:0/3023849935 wait complete. 2024-02-19T22:17:33.953 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:33.196730+0000 mgr.smithi111.ftjzov (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:33.949701+0000 mon.smithi111 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.203:0/3023849935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:35.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:35.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:35.197047+0000 mgr.smithi111.ftjzov (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:37.598 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- 172.21.15.203:0/2257237781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 msgr2=0x7ff0c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/2257237781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c00f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7ff0b0009fb0 tx=0x7ff0b0009b30).stop 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- 172.21.15.203:0/2257237781 shutdown_connections 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/2257237781 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c00f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- 172.21.15.203:0/2257237781 >> 172.21.15.203:0/2257237781 conn(0x7ff0c00f09a0 msgr2=0x7ff0c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- 172.21.15.203:0/2257237781 shutdown_connections 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- 172.21.15.203:0/2257237781 wait complete. 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 Processor -- start 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.596+0000 7ff0bf59e700 1 -- start start 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0bf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0bf59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c0106ac0 con 0x7ff0c00f51a0 2024-02-19T22:17:37.599 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0be59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:40496/0 (socket says 172.21.15.203:40496) 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0be59c700 1 -- 172.21.15.203:0/488023457 learned_addr learned my addr 172.21.15.203:0/488023457 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0be59c700 1 -- 172.21.15.203:0/488023457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0005040 con 0x7ff0c00f51a0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0be59c700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff0b00089f0 tx=0x7ff0b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b0009320 con 0x7ff0c00f51a0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff0b0013380 con 0x7ff0c00f51a0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0b0014e20 con 0x7ff0c00f51a0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.597+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c0106cc0 con 0x7ff0c00f51a0 2024-02-19T22:17:37.600 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.598+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c0107150 con 0x7ff0c00f51a0 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.598+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0b00095f0 con 0x7ff0c00f51a0 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.598+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c0045ad0 con 0x7ff0c00f51a0 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.598+0000 7ff0aeffd700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 0x7ff0a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.598+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0b0040f10 con 0x7ff0c00f51a0 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.599+0000 7ff0bdd9b700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 0x7ff0a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:37.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.599+0000 7ff0bdd9b700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 0x7ff0a80323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff0b40096a0 tx=0x7ff0b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:37.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.603+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0b0014420 con 0x7ff0c00f51a0 2024-02-19T22:17:37.811 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.810+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0c00fad30 con 0x7ff0c00f51a0 2024-02-19T22:17:37.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.811+0000 7ff0aeffd700 1 -- 172.21.15.203:0/488023457 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff0b001d980 con 0x7ff0c00f51a0 2024-02-19T22:17:37.812 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:37.812 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 msgr2=0x7ff0a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 0x7ff0a80323d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff0b40096a0 tx=0x7ff0b4006b40).stop 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 msgr2=0x7ff0c01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7ff0b00089f0 tx=0x7ff0b000b040).stop 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 shutdown_connections 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff0a802ff10 0x7ff0a80323d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 --2- 172.21.15.203:0/488023457 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff0c00f51a0 0x7ff0c01065d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:37.814 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 >> 172.21.15.203:0/488023457 conn(0x7ff0c00f09a0 msgr2=0x7ff0c005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:37.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 shutdown_connections 2024-02-19T22:17:37.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:37.813+0000 7ff0bf59e700 1 -- 172.21.15.203:0/488023457 wait complete. 2024-02-19T22:17:37.815 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:37.197397+0000 mgr.smithi111.ftjzov (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:37.811058+0000 mon.smithi111 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.203:0/488023457' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:39.513 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:39.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:39.197714+0000 mgr.smithi111.ftjzov (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- 172.21.15.203:0/3448087891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 msgr2=0x7f76a80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3448087891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a80f55c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f7698009fb0 tx=0x7f7698009b30).stop 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- 172.21.15.203:0/3448087891 shutdown_connections 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3448087891 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a80f55c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- 172.21.15.203:0/3448087891 >> 172.21.15.203:0/3448087891 conn(0x7f76a80f09a0 msgr2=0x7f76a80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- 172.21.15.203:0/3448087891 shutdown_connections 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- 172.21.15.203:0/3448087891 wait complete. 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 Processor -- start 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- start start 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:41.485 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.483+0000 7f76aef3d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a8106ad0 con 0x7f76a80f51a0 2024-02-19T22:17:41.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76adf3b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:41.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76adf3b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:40512/0 (socket says 172.21.15.203:40512) 2024-02-19T22:17:41.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76adf3b700 1 -- 172.21.15.203:0/3798004685 learned_addr learned my addr 172.21.15.203:0/3798004685 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:41.486 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76adf3b700 1 -- 172.21.15.203:0/3798004685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7698005040 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76adf3b700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f76980089f0 tx=0x7f769800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7698009320 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a8106cd0 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7698013380 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7698014e20 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.484+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a8107160 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.485+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76980095f0 con 0x7f76a80f51a0 2024-02-19T22:17:41.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.485+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a8045ad0 con 0x7f76a80f51a0 2024-02-19T22:17:41.488 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.485+0000 7f769e7fc700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 0x7f76940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:41.488 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.485+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7698040e90 con 0x7f76a80f51a0 2024-02-19T22:17:41.488 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.486+0000 7f76ad73a700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 0x7f76940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:41.488 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.486+0000 7f76ad73a700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 0x7f76940323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f76a40096a0 tx=0x7f76a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:41.491 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.490+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7698014420 con 0x7f76a80f51a0 2024-02-19T22:17:41.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.694+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76a8108010 con 0x7f76a80f51a0 2024-02-19T22:17:41.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.695+0000 7f769e7fc700 1 -- 172.21.15.203:0/3798004685 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f769801d980 con 0x7f76a80f51a0 2024-02-19T22:17:41.696 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:41.696 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.696+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 msgr2=0x7f76940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 0x7f76940323d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f76a40096a0 tx=0x7f76a4006b40).stop 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 msgr2=0x7f76a81065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f76980089f0 tx=0x7f769800b040).stop 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 shutdown_connections 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f769402ff10 0x7f76940323d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 --2- 172.21.15.203:0/3798004685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76a80f51a0 0x7f76a81065e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:41.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 >> 172.21.15.203:0/3798004685 conn(0x7f76a80f09a0 msgr2=0x7f76a80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:41.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 shutdown_connections 2024-02-19T22:17:41.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:41.697+0000 7f76aef3d700 1 -- 172.21.15.203:0/3798004685 wait complete. 2024-02-19T22:17:41.699 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:41.198057+0000 mgr.smithi111.ftjzov (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:41.695204+0000 mon.smithi111 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.203:0/3798004685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:43.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:43.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:43.198384+0000 mgr.smithi111.ftjzov (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:45.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 -- 172.21.15.203:0/100052354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 msgr2=0x7f8b740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:45.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 --2- 172.21.15.203:0/100052354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b740f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f8b68009fb0 tx=0x7f8b68009b30).stop 2024-02-19T22:17:45.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 -- 172.21.15.203:0/100052354 shutdown_connections 2024-02-19T22:17:45.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 --2- 172.21.15.203:0/100052354 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b740f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:45.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 -- 172.21.15.203:0/100052354 >> 172.21.15.203:0/100052354 conn(0x7f8b740f0b70 msgr2=0x7f8b740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:45.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 -- 172.21.15.203:0/100052354 shutdown_connections 2024-02-19T22:17:45.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.498+0000 7f8b7a286700 1 -- 172.21.15.203:0/100052354 wait complete. 2024-02-19T22:17:45.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b7a286700 1 Processor -- start 2024-02-19T22:17:45.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b7a286700 1 -- start start 2024-02-19T22:17:45.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b7a286700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:45.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b7a286700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b74060240 con 0x7f8b740f5410 2024-02-19T22:17:45.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b73fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37308/0 (socket says 172.21.15.203:37308) 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.499+0000 7f8b73fff700 1 -- 172.21.15.203:0/768095491 learned_addr learned my addr 172.21.15.203:0/768095491 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b73fff700 1 -- 172.21.15.203:0/768095491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68005040 con 0x7f8b740f5410 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b73fff700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f8b68005010 tx=0x7f8b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b68009320 con 0x7f8b740f5410 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b74060440 con 0x7f8b740f5410 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8b68013380 con 0x7f8b740f5410 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b68014e50 con 0x7f8b740f5410 2024-02-19T22:17:45.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.500+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b74109b90 con 0x7f8b740f5410 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.501+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b680263a0 con 0x7f8b740f5410 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.501+0000 7f8b70ff9700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 0x7f8b5c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.501+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b68040f70 con 0x7f8b740f5410 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.501+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b74045ad0 con 0x7f8b740f5410 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.501+0000 7f8b737fe700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 0x7f8b5c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:45.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.502+0000 7f8b737fe700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 0x7f8b5c0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8b640096a0 tx=0x7f8b64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:45.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.506+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b68012920 con 0x7f8b740f5410 2024-02-19T22:17:45.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.712+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b740fafa0 con 0x7f8b740f5410 2024-02-19T22:17:45.715 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.713+0000 7f8b70ff9700 1 -- 172.21.15.203:0/768095491 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b6802b020 con 0x7f8b740f5410 2024-02-19T22:17:45.715 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:45.715 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 msgr2=0x7f8b5c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 0x7f8b5c0323d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f8b640096a0 tx=0x7f8b64006b40).stop 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 msgr2=0x7f8b741094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f8b68005010 tx=0x7f8b6800b040).stop 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 shutdown_connections 2024-02-19T22:17:45.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8b5c02ff10 0x7f8b5c0323d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:45.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 --2- 172.21.15.203:0/768095491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8b740f5410 0x7f8b741094e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:45.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 >> 172.21.15.203:0/768095491 conn(0x7f8b740f0b70 msgr2=0x7f8b740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:45.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 shutdown_connections 2024-02-19T22:17:45.718 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:45.715+0000 7f8b7a286700 1 -- 172.21.15.203:0/768095491 wait complete. 2024-02-19T22:17:45.718 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:45.198701+0000 mgr.smithi111.ftjzov (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:45.714432+0000 mon.smithi111 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.203:0/768095491' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:47.294 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:47.295 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:47.198991+0000 mgr.smithi111.ftjzov (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:49.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 -- 172.21.15.203:0/1917862565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 msgr2=0x7f7a000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/1917862565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a000f5830 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f79f4009fb0 tx=0x7f79f4009b30).stop 2024-02-19T22:17:49.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 -- 172.21.15.203:0/1917862565 shutdown_connections 2024-02-19T22:17:49.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/1917862565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a000f5830 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 -- 172.21.15.203:0/1917862565 >> 172.21.15.203:0/1917862565 conn(0x7f7a000f0b70 msgr2=0x7f7a000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 -- 172.21.15.203:0/1917862565 shutdown_connections 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.188+0000 7f7a06bf2700 1 -- 172.21.15.203:0/1917862565 wait complete. 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a06bf2700 1 Processor -- start 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a06bf2700 1 -- start start 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a06bf2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a06bf2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a00106cd0 con 0x7f7a000f5410 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a0498e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a0498e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37322/0 (socket says 172.21.15.203:37322) 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.189+0000 7f7a0498e700 1 -- 172.21.15.203:0/2331741810 learned_addr learned my addr 172.21.15.203:0/2331741810 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f7a0498e700 1 -- 172.21.15.203:0/2331741810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f4005040 con 0x7f7a000f5410 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f7a0498e700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f79f4005010 tx=0x7f79f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79f4009320 con 0x7f7a000f5410 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79f4013380 con 0x7f7a000f5410 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a00106ed0 con 0x7f7a000f5410 2024-02-19T22:17:49.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79f4014e50 con 0x7f7a000f5410 2024-02-19T22:17:49.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.190+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a00107360 con 0x7f7a000f5410 2024-02-19T22:17:49.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.191+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79f40095f0 con 0x7f7a000f5410 2024-02-19T22:17:49.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.191+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a00045ad0 con 0x7f7a000f5410 2024-02-19T22:17:49.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.191+0000 7f79fd7fa700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 0x7f79f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:49.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.191+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79f4040e90 con 0x7f7a000f5410 2024-02-19T22:17:49.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.191+0000 7f79fffff700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 0x7f79f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:49.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.192+0000 7f79fffff700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 0x7f79f00323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f79ec0096a0 tx=0x7f79ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:49.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.196+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79f4014420 con 0x7f7a000f5410 2024-02-19T22:17:49.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.402+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a000fafa0 con 0x7f7a000f5410 2024-02-19T22:17:49.404 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.402+0000 7f79fd7fa700 1 -- 172.21.15.203:0/2331741810 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f79f4010230 con 0x7f7a000f5410 2024-02-19T22:17:49.404 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:49.405 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 msgr2=0x7f79f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 0x7f79f00323d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f79ec0096a0 tx=0x7f79ec006b40).stop 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 msgr2=0x7f7a001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f79f4005010 tx=0x7f79f400b040).stop 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 shutdown_connections 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f79f002ff10 0x7f79f00323d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 --2- 172.21.15.203:0/2331741810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7a000f5410 0x7f7a001067e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:49.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 >> 172.21.15.203:0/2331741810 conn(0x7f7a000f0b70 msgr2=0x7f7a0005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:49.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 shutdown_connections 2024-02-19T22:17:49.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:49.404+0000 7f7a06bf2700 1 -- 172.21.15.203:0/2331741810 wait complete. 2024-02-19T22:17:49.407 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:49.199258+0000 mgr.smithi111.ftjzov (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:50.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:49.403734+0000 mon.smithi111 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.203:0/2331741810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:51.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:51.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:52.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:51.199592+0000 mgr.smithi111.ftjzov (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.922+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3010689287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 msgr2=0x7f2fa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.922+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3010689287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa00f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f2f90009fb0 tx=0x7f2f90009b30).stop 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3010689287 shutdown_connections 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3010689287 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa00f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3010689287 >> 172.21.15.203:0/3010689287 conn(0x7f2fa00f0b70 msgr2=0x7f2fa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:52.925 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3010689287 shutdown_connections 2024-02-19T22:17:52.926 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3010689287 wait complete. 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 Processor -- start 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.923+0000 7f2fa7ca5700 1 -- start start 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa7ca5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa7ca5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa0060240 con 0x7f2fa00f5410 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa5a41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa5a41700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37336/0 (socket says 172.21.15.203:37336) 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa5a41700 1 -- 172.21.15.203:0/3576646892 learned_addr learned my addr 172.21.15.203:0/3576646892 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa5a41700 1 -- 172.21.15.203:0/3576646892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f90005040 con 0x7f2fa00f5410 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa5a41700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f2f90005010 tx=0x7f2f9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:52.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f90009320 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa0060440 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f90013380 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f90014e50 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.924+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa0109b90 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.925+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f900263a0 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.925+0000 7f2f967fc700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 0x7f2f8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.925+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f90040ea0 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.925+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa0045ad0 con 0x7f2fa00f5410 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.926+0000 7f2fa5240700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 0x7f2f8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:52.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.926+0000 7f2fa5240700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 0x7f2f8c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2f9c0096a0 tx=0x7f2f9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:52.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:52.930+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f90012920 con 0x7f2fa00f5410 2024-02-19T22:17:53.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.137+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fa010a300 con 0x7f2fa00f5410 2024-02-19T22:17:53.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.138+0000 7f2f967fc700 1 -- 172.21.15.203:0/3576646892 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f90012330 con 0x7f2fa00f5410 2024-02-19T22:17:53.140 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:53.140 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 msgr2=0x7f2f8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 0x7f2f8c0323d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f2f9c0096a0 tx=0x7f2f9c006b40).stop 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 msgr2=0x7f2fa01094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f2f90005010 tx=0x7f2f9000b040).stop 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 shutdown_connections 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2f8c02ff10 0x7f2f8c0323d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 --2- 172.21.15.203:0/3576646892 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2fa00f5410 0x7f2fa01094e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 >> 172.21.15.203:0/3576646892 conn(0x7f2fa00f0b70 msgr2=0x7f2fa00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 shutdown_connections 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:53.139+0000 7f2fa7ca5700 1 -- 172.21.15.203:0/3576646892 wait complete. 2024-02-19T22:17:53.142 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:53.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:53.139155+0000 mon.smithi111 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.203:0/3576646892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:54.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:53.199968+0000 mgr.smithi111.ftjzov (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:54.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:54.778 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.637+0000 7faf2f59e700 1 -- 172.21.15.203:0/3003064309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 msgr2=0x7faf300f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.637+0000 7faf2f59e700 1 --2- 172.21.15.203:0/3003064309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf300f66c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7faf20009fb0 tx=0x7faf20009b30).stop 2024-02-19T22:17:56.641 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- 172.21.15.203:0/3003064309 shutdown_connections 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 --2- 172.21.15.203:0/3003064309 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf300f66c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- 172.21.15.203:0/3003064309 >> 172.21.15.203:0/3003064309 conn(0x7faf300f1aa0 msgr2=0x7faf300f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- 172.21.15.203:0/3003064309 shutdown_connections 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- 172.21.15.203:0/3003064309 wait complete. 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 Processor -- start 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- start start 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2f59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf30106ac0 con 0x7faf300f62a0 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.638+0000 7faf2e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50956/0 (socket says 172.21.15.203:50956) 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2e59c700 1 -- 172.21.15.203:0/169035312 learned_addr learned my addr 172.21.15.203:0/169035312 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:17:56.642 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2e59c700 1 -- 172.21.15.203:0/169035312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf20005040 con 0x7faf300f62a0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2e59c700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7faf200089f0 tx=0x7faf2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf20009320 con 0x7faf300f62a0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf30106cc0 con 0x7faf300f62a0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf20013380 con 0x7faf300f62a0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf30107150 con 0x7faf300f62a0 2024-02-19T22:17:56.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.639+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf20014e20 con 0x7faf300f62a0 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.640+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7faf200263a0 con 0x7faf300f62a0 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.640+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf30045ad0 con 0x7faf300f62a0 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.640+0000 7faf26ffd700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 0x7faf1c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.640+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7faf20040f10 con 0x7faf300f62a0 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.640+0000 7faf2dd9b700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 0x7faf1c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:17:56.644 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.641+0000 7faf2dd9b700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 0x7faf1c0323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7faf180096a0 tx=0x7faf18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:17:56.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.645+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf200126f0 con 0x7faf300f62a0 2024-02-19T22:17:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:55.200326+0000 mgr.smithi111.ftjzov (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:17:56.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.849+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf30107fa0 con 0x7faf300f62a0 2024-02-19T22:17:56.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.850+0000 7faf26ffd700 1 -- 172.21.15.203:0/169035312 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7faf2000f340 con 0x7faf300f62a0 2024-02-19T22:17:56.853 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:17:56.853 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:17:56.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 msgr2=0x7faf1c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:56.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 0x7faf1c0323d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7faf180096a0 tx=0x7faf18006b40).stop 2024-02-19T22:17:56.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 msgr2=0x7faf301065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:17:56.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7faf200089f0 tx=0x7faf2000b040).stop 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 shutdown_connections 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7faf1c02ff10 0x7faf1c0323d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 --2- 172.21.15.203:0/169035312 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faf300f62a0 0x7faf301065d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 >> 172.21.15.203:0/169035312 conn(0x7faf300f1aa0 msgr2=0x7faf300f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 shutdown_connections 2024-02-19T22:17:56.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:17:56.852+0000 7faf2f59e700 1 -- 172.21.15.203:0/169035312 wait complete. 2024-02-19T22:17:56.856 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:17:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:17:56.851424+0000 mon.smithi111 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.203:0/169035312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:17:58.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:17:58.515 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:17:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:17:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:57.200719+0000 mgr.smithi111.ftjzov (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 -- 172.21.15.203:0/2588544108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 msgr2=0x7f3ae00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2588544108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae00f6930 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3ad4009fb0 tx=0x7f3ad4009b30).stop 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 -- 172.21.15.203:0/2588544108 shutdown_connections 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2588544108 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae00f6930 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 -- 172.21.15.203:0/2588544108 >> 172.21.15.203:0/2588544108 conn(0x7f3ae00f1c70 msgr2=0x7f3ae00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 -- 172.21.15.203:0/2588544108 shutdown_connections 2024-02-19T22:18:00.370 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.367+0000 7f3ae5036700 1 -- 172.21.15.203:0/2588544108 wait complete. 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.368+0000 7f3ae5036700 1 Processor -- start 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.368+0000 7f3ae5036700 1 -- start start 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ae5036700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ae5036700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae0106cd0 con 0x7f3ae00f6510 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3aded9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3aded9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50978/0 (socket says 172.21.15.203:50978) 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3aded9d700 1 -- 172.21.15.203:0/2545501852 learned_addr learned my addr 172.21.15.203:0/2545501852 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3aded9d700 1 -- 172.21.15.203:0/2545501852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4005040 con 0x7f3ae00f6510 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3aded9d700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3ad4005010 tx=0x7f3ad400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:00.372 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad4009320 con 0x7f3ae00f6510 2024-02-19T22:18:00.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ad4013380 con 0x7f3ae00f6510 2024-02-19T22:18:00.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae0106ed0 con 0x7f3ae00f6510 2024-02-19T22:18:00.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ad4014e50 con 0x7f3ae00f6510 2024-02-19T22:18:00.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.369+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae0107360 con 0x7f3ae00f6510 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.370+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ad40095f0 con 0x7f3ae00f6510 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.370+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0045ad0 con 0x7f3ae00f6510 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.370+0000 7f3ac77fe700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 0x7f3ac80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.370+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ad4040e90 con 0x7f3ae00f6510 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.371+0000 7f3ade59c700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 0x7f3ac80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:00.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.371+0000 7f3ade59c700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 0x7f3ac80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3ad00096a0 tx=0x7f3ad0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:00.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.375+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ad4014420 con 0x7f3ae00f6510 2024-02-19T22:18:00.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.580+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ae0108210 con 0x7f3ae00f6510 2024-02-19T22:18:00.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.581+0000 7f3ac77fe700 1 -- 172.21.15.203:0/2545501852 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ad4010230 con 0x7f3ae00f6510 2024-02-19T22:18:00.583 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:00.583 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:00.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 msgr2=0x7f3ac80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:00.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 0x7f3ac80323d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3ad00096a0 tx=0x7f3ad0006b40).stop 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 msgr2=0x7f3ae01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3ad4005010 tx=0x7f3ad400b040).stop 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 shutdown_connections 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3ac802ff10 0x7f3ac80323d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 --2- 172.21.15.203:0/2545501852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3ae00f6510 0x7f3ae01067e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 >> 172.21.15.203:0/2545501852 conn(0x7f3ae00f1c70 msgr2=0x7f3ae00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 shutdown_connections 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:00.582+0000 7f3ae5036700 1 -- 172.21.15.203:0/2545501852 wait complete. 2024-02-19T22:18:00.585 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:17:59.201099+0000 mgr.smithi111.ftjzov (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:01.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:00.581862+0000 mon.smithi111 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.203:0/2545501852' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:02.283 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:02.284 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:01.201469+0000 mgr.smithi111.ftjzov (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 -- 172.21.15.203:0/495089815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 msgr2=0x7fdb380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 --2- 172.21.15.203:0/495089815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb380f5830 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fdb28009fb0 tx=0x7fdb28009b30).stop 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 -- 172.21.15.203:0/495089815 shutdown_connections 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 --2- 172.21.15.203:0/495089815 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb380f5830 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 -- 172.21.15.203:0/495089815 >> 172.21.15.203:0/495089815 conn(0x7fdb380f0b70 msgr2=0x7fdb380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:04.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 -- 172.21.15.203:0/495089815 shutdown_connections 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.242+0000 7fdb40297700 1 -- 172.21.15.203:0/495089815 wait complete. 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb40297700 1 Processor -- start 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb40297700 1 -- start start 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb40297700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb40297700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38106d00 con 0x7fdb380f5410 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb3e033700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb3e033700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:50994/0 (socket says 172.21.15.203:50994) 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb3e033700 1 -- 172.21.15.203:0/3484294742 learned_addr learned my addr 172.21.15.203:0/3484294742 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb3e033700 1 -- 172.21.15.203:0/3484294742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb28005040 con 0x7fdb380f5410 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.243+0000 7fdb3e033700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdb28005010 tx=0x7fdb2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28009320 con 0x7fdb380f5410 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb28013380 con 0x7fdb380f5410 2024-02-19T22:18:04.247 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb38106f00 con 0x7fdb380f5410 2024-02-19T22:18:04.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb28014e50 con 0x7fdb380f5410 2024-02-19T22:18:04.248 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb38107390 con 0x7fdb380f5410 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb280095f0 con 0x7fdb380f5410 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb38045ad0 con 0x7fdb380f5410 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.244+0000 7fdb2effd700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 0x7fdb240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.245+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb28040fa0 con 0x7fdb380f5410 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.245+0000 7fdb3d832700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 0x7fdb240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:04.249 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.245+0000 7fdb3d832700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 0x7fdb240323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdb340096a0 tx=0x7fdb34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:04.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.249+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb28014420 con 0x7fdb380f5410 2024-02-19T22:18:04.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.454+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb380fafa0 con 0x7fdb380f5410 2024-02-19T22:18:04.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.454+0000 7fdb2effd700 1 -- 172.21.15.203:0/3484294742 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb28010230 con 0x7fdb380f5410 2024-02-19T22:18:04.457 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:04.457 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:04.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 msgr2=0x7fdb240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 0x7fdb240323d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdb340096a0 tx=0x7fdb34006b40).stop 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 msgr2=0x7fdb38106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fdb28005010 tx=0x7fdb2800b040).stop 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 shutdown_connections 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb2402ff10 0x7fdb240323d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 --2- 172.21.15.203:0/3484294742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb380f5410 0x7fdb38106810 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 >> 172.21.15.203:0/3484294742 conn(0x7fdb380f0b70 msgr2=0x7fdb3805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 shutdown_connections 2024-02-19T22:18:04.459 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:04.456+0000 7fdb40297700 1 -- 172.21.15.203:0/3484294742 wait complete. 2024-02-19T22:18:04.460 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:03.201748+0000 mgr.smithi111.ftjzov (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:05.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:04.455711+0000 mon.smithi111 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.203:0/3484294742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:06.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:06.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:05.202096+0000 mgr.smithi111.ftjzov (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:08.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 -- 172.21.15.203:0/1706838474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 msgr2=0x7fdba80f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 --2- 172.21.15.203:0/1706838474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f8f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fdb98009fb0 tx=0x7fdb98009b30).stop 2024-02-19T22:18:08.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 -- 172.21.15.203:0/1706838474 shutdown_connections 2024-02-19T22:18:08.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 --2- 172.21.15.203:0/1706838474 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f8f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 -- 172.21.15.203:0/1706838474 >> 172.21.15.203:0/1706838474 conn(0x7fdba80f09a0 msgr2=0x7fdba80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:08.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 -- 172.21.15.203:0/1706838474 shutdown_connections 2024-02-19T22:18:08.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 -- 172.21.15.203:0/1706838474 wait complete. 2024-02-19T22:18:08.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.150+0000 7fdba759e700 1 Processor -- start 2024-02-19T22:18:08.158 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba759e700 1 -- start start 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba80f4e20 con 0x7fdba80601e0 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55530/0 (socket says 172.21.15.203:55530) 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba659c700 1 -- 172.21.15.203:0/1701690931 learned_addr learned my addr 172.21.15.203:0/1701690931 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:08.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba659c700 1 -- 172.21.15.203:0/1701690931 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb98005040 con 0x7fdba80601e0 2024-02-19T22:18:08.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba659c700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdb98005010 tx=0x7fdb9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:08.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb98009320 con 0x7fdba80601e0 2024-02-19T22:18:08.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba80f4fc0 con 0x7fdba80601e0 2024-02-19T22:18:08.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.151+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb98013380 con 0x7fdba80601e0 2024-02-19T22:18:08.160 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb98014e50 con 0x7fdba80601e0 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba80f5450 con 0x7fdba80601e0 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb980095f0 con 0x7fdba80601e0 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba8059100 con 0x7fdba80601e0 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdb96ffd700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 0x7fdb900323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.152+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb98040fa0 con 0x7fdba80601e0 2024-02-19T22:18:08.161 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.153+0000 7fdba5d9b700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 0x7fdb900323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:08.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.153+0000 7fdba5d9b700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 0x7fdb900323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdb9c0096a0 tx=0x7fdb9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:08.162 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.157+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb98014420 con 0x7fdba80601e0 2024-02-19T22:18:08.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.362+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdba80fbe10 con 0x7fdba80601e0 2024-02-19T22:18:08.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdb96ffd700 1 -- 172.21.15.203:0/1701690931 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb98010230 con 0x7fdba80601e0 2024-02-19T22:18:08.365 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:08.365 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:08.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 msgr2=0x7fdb900323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 0x7fdb900323d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdb9c0096a0 tx=0x7fdb9c006b40).stop 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 msgr2=0x7fdba80f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fdb98005010 tx=0x7fdb9800b040).stop 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 shutdown_connections 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb9002ff10 0x7fdb900323d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 --2- 172.21.15.203:0/1701690931 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdba80601e0 0x7fdba80f69a0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 >> 172.21.15.203:0/1701690931 conn(0x7fdba80f09a0 msgr2=0x7fdba80f13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 shutdown_connections 2024-02-19T22:18:08.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:08.364+0000 7fdba759e700 1 -- 172.21.15.203:0/1701690931 wait complete. 2024-02-19T22:18:08.368 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:07.202423+0000 mgr.smithi111.ftjzov (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:09.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:09 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:08.364127+0000 mon.smithi111 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.203:0/1701690931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:09.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:09.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:09.202702+0000 mgr.smithi111.ftjzov (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 -- 172.21.15.203:0/3797205289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 msgr2=0x7f628c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 --2- 172.21.15.203:0/3797205289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c0f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f627c009fb0 tx=0x7f627c009b30).stop 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 -- 172.21.15.203:0/3797205289 shutdown_connections 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 --2- 172.21.15.203:0/3797205289 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c0f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 -- 172.21.15.203:0/3797205289 >> 172.21.15.203:0/3797205289 conn(0x7f628c0f1c70 msgr2=0x7f628c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 -- 172.21.15.203:0/3797205289 shutdown_connections 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.841+0000 7f62922b7700 1 -- 172.21.15.203:0/3797205289 wait complete. 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f62922b7700 1 Processor -- start 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f62922b7700 1 -- start start 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f62922b7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:11.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f62922b7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c106cd0 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f628bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f628bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55554/0 (socket says 172.21.15.203:55554) 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.842+0000 7f628bfff700 1 -- 172.21.15.203:0/1221302499 learned_addr learned my addr 172.21.15.203:0/1221302499 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f628bfff700 1 -- 172.21.15.203:0/1221302499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627c005040 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f628bfff700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f627c005010 tx=0x7f627c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627c009320 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c106ed0 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f627c013380 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f627c014e50 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.843+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c107360 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.844+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f627c014420 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.844+0000 7f6288ff9700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 0x7f62740323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.844+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f627c040f40 con 0x7f628c0f6510 2024-02-19T22:18:11.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.844+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c045ad0 con 0x7f628c0f6510 2024-02-19T22:18:11.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.844+0000 7f628b7fe700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 0x7f62740323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:11.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.845+0000 7f628b7fe700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 0x7f62740323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f62800096a0 tx=0x7f6280006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:11.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:11.849+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f627c0095f0 con 0x7f628c0f6510 2024-02-19T22:18:12.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.053+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f628c0fc190 con 0x7f628c0f6510 2024-02-19T22:18:12.056 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.054+0000 7f6288ff9700 1 -- 172.21.15.203:0/1221302499 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f627c02b020 con 0x7f628c0f6510 2024-02-19T22:18:12.056 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:12.056 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.055+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 msgr2=0x7f62740323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.055+0000 7f62922b7700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 0x7f62740323d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f62800096a0 tx=0x7f6280006b40).stop 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.055+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 msgr2=0x7f628c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.055+0000 7f62922b7700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f627c005010 tx=0x7f627c00b040).stop 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.055+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 shutdown_connections 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.056+0000 7f62922b7700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f627402ff10 0x7f62740323d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.056+0000 7f62922b7700 1 --2- 172.21.15.203:0/1221302499 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f628c0f6510 0x7f628c1067e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.056+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 >> 172.21.15.203:0/1221302499 conn(0x7f628c0f1c70 msgr2=0x7f628c0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.056+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 shutdown_connections 2024-02-19T22:18:12.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:12.056+0000 7f62922b7700 1 -- 172.21.15.203:0/1221302499 wait complete. 2024-02-19T22:18:12.059 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:12.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:11.202947+0000 mgr.smithi111.ftjzov (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:12.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:12.055177+0000 mon.smithi111 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.203:0/1221302499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:13.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:13.703 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:13.203303+0000 mgr.smithi111.ftjzov (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:15.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 -- 172.21.15.203:0/2678182516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 msgr2=0x7f81240f6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:15.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 --2- 172.21.15.203:0/2678182516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f81240f6da0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f8118009fb0 tx=0x7f8118009b30).stop 2024-02-19T22:18:15.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 -- 172.21.15.203:0/2678182516 shutdown_connections 2024-02-19T22:18:15.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 --2- 172.21.15.203:0/2678182516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f81240f6da0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:15.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 -- 172.21.15.203:0/2678182516 >> 172.21.15.203:0/2678182516 conn(0x7f81240f0260 msgr2=0x7f81240f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 -- 172.21.15.203:0/2678182516 shutdown_connections 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.517+0000 7f812acb9700 1 -- 172.21.15.203:0/2678182516 wait complete. 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f812acb9700 1 Processor -- start 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f812acb9700 1 -- start start 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f812acb9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:15.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f812acb9700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81240f5270 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f8128a55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f8128a55700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:45776/0 (socket says 172.21.15.203:45776) 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.518+0000 7f8128a55700 1 -- 172.21.15.203:0/2932717057 learned_addr learned my addr 172.21.15.203:0/2932717057 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f8128a55700 1 -- 172.21.15.203:0/2932717057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8118005040 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f8128a55700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8118005010 tx=0x7f811800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118009320 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81240f5470 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8118013380 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8118014e50 con 0x7f81240f6980 2024-02-19T22:18:15.522 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.519+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8124109bb0 con 0x7f81240f6980 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8118014420 con 0x7f81240f6980 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f81217fa700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 0x7f81140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8124045ad0 con 0x7f81240f6980 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8118040e30 con 0x7f81240f6980 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f8123fff700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 0x7f81140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:15.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.520+0000 7f8123fff700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 0x7f81140323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f81100096a0 tx=0x7f8110006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:15.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.525+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f81180095f0 con 0x7f81240f6980 2024-02-19T22:18:15.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.729+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f812410a3d0 con 0x7f81240f6980 2024-02-19T22:18:15.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.730+0000 7f81217fa700 1 -- 172.21.15.203:0/2932717057 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8118040070 con 0x7f81240f6980 2024-02-19T22:18:15.731 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:15.731 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 msgr2=0x7f81140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 0x7f81140323d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f81100096a0 tx=0x7f8110006b40).stop 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 msgr2=0x7f8124109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f8118005010 tx=0x7f811800b040).stop 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 shutdown_connections 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f811402ff10 0x7f81140323d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 --2- 172.21.15.203:0/2932717057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f81240f6980 0x7f8124109500 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 >> 172.21.15.203:0/2932717057 conn(0x7f81240f0260 msgr2=0x7f81240f0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 shutdown_connections 2024-02-19T22:18:15.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:15.731+0000 7f812acb9700 1 -- 172.21.15.203:0/2932717057 wait complete. 2024-02-19T22:18:15.734 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:15.203512+0000 mgr.smithi111.ftjzov (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:15.729888+0000 mon.smithi111 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.203:0/2932717057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:17.300 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:17.301 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:17.203759+0000 mgr.smithi111.ftjzov (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:17.300043+0000 mon.smithi111 (mon.0) 195 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:18:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:17.302380+0000 mon.smithi111 (mon.0) 196 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:18:19.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.290+0000 7f2d4bfff700 1 -- 172.21.15.203:0/3821475592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 msgr2=0x7f2d4c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.290+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/3821475592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c0f66c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f2d3c009fb0 tx=0x7f2d3c009b30).stop 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 -- 172.21.15.203:0/3821475592 shutdown_connections 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/3821475592 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c0f66c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 -- 172.21.15.203:0/3821475592 >> 172.21.15.203:0/3821475592 conn(0x7f2d4c0f1aa0 msgr2=0x7f2d4c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 -- 172.21.15.203:0/3821475592 shutdown_connections 2024-02-19T22:18:19.293 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 -- 172.21.15.203:0/3821475592 wait complete. 2024-02-19T22:18:19.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 Processor -- start 2024-02-19T22:18:19.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.291+0000 7f2d4bfff700 1 -- start start 2024-02-19T22:18:19.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:19.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4c106ac0 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:45800/0 (socket says 172.21.15.203:45800) 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4affd700 1 -- 172.21.15.203:0/2421146744 learned_addr learned my addr 172.21.15.203:0/2421146744 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4affd700 1 -- 172.21.15.203:0/2421146744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c005040 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d4affd700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2d3c009830 tx=0x7f2d3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.292+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3c009320 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d4c106cc0 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d3c013380 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d3c014420 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d4c107150 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d3c014580 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d337fe700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.293+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d3c041fa0 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.294+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d4c059100 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.294+0000 7f2d4a7fc700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:19.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.294+0000 7f2d4a7fc700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 0x7f2d34032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f2d400096a0 tx=0x7f2d40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:19.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.298+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d3c0273f0 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.505+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d4c045ad0 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.505+0000 7f2d337fe700 1 -- 172.21.15.203:0/2421146744 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2d3c01c890 con 0x7f2d4c0f62a0 2024-02-19T22:18:19.507 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:19.507 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:19.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 msgr2=0x7f2d34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 0x7f2d34032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f2d400096a0 tx=0x7f2d40006b40).stop 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 msgr2=0x7f2d4c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f2d3c009830 tx=0x7f2d3c00b040).stop 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 shutdown_connections 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d3402ff60 0x7f2d34032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 --2- 172.21.15.203:0/2421146744 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2d4c0f62a0 0x7f2d4c1065d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 >> 172.21.15.203:0/2421146744 conn(0x7f2d4c0f1aa0 msgr2=0x7f2d4c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 shutdown_connections 2024-02-19T22:18:19.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:19.507+0000 7f2d4bfff700 1 -- 172.21.15.203:0/2421146744 wait complete. 2024-02-19T22:18:19.510 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:19.204124+0000 mgr.smithi111.ftjzov (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:20.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:19.505719+0000 mon.smithi111 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.203:0/2421146744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:21.136 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:21.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:22.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:21.204363+0000 mgr.smithi111.ftjzov (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 -- 172.21.15.203:0/2657762973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 msgr2=0x7f57d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 --2- 172.21.15.203:0/2657762973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d40f6930 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f57c4009fb0 tx=0x7f57c4009b30).stop 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 -- 172.21.15.203:0/2657762973 shutdown_connections 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 --2- 172.21.15.203:0/2657762973 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d40f6930 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 -- 172.21.15.203:0/2657762973 >> 172.21.15.203:0/2657762973 conn(0x7f57d40f1c70 msgr2=0x7f57d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 -- 172.21.15.203:0/2657762973 shutdown_connections 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.160+0000 7f57db564700 1 -- 172.21.15.203:0/2657762973 wait complete. 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.161+0000 7f57db564700 1 Processor -- start 2024-02-19T22:18:23.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.161+0000 7f57db564700 1 -- start start 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.161+0000 7f57db564700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.161+0000 7f57db564700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d4069e20 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57d9300700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57d9300700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:45818/0 (socket says 172.21.15.203:45818) 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57d9300700 1 -- 172.21.15.203:0/43988531 learned_addr learned my addr 172.21.15.203:0/43988531 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57d9300700 1 -- 172.21.15.203:0/43988531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4005040 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57d9300700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f57c4005010 tx=0x7f57c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c4009320 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d4069fc0 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57c4013380 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57c4014e50 con 0x7f57d40f6510 2024-02-19T22:18:23.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.162+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d406a450 con 0x7f57d40f6510 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.163+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57c40095f0 con 0x7f57d40f6510 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.163+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d4059100 con 0x7f57d40f6510 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.163+0000 7f57c9ffb700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 0x7f57c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.163+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57c4040f80 con 0x7f57d40f6510 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.164+0000 7f57d8aff700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 0x7f57c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.164+0000 7f57d8aff700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 0x7f57c00323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f57d00096a0 tx=0x7f57d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:23.277 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.168+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57c4014420 con 0x7f57d40f6510 2024-02-19T22:18:23.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.376+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57d40fc0a0 con 0x7f57d40f6510 2024-02-19T22:18:23.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.377+0000 7f57c9ffb700 1 -- 172.21.15.203:0/43988531 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57c4010230 con 0x7f57d40f6510 2024-02-19T22:18:23.380 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:23.380 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 msgr2=0x7f57c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 0x7f57c00323d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f57d00096a0 tx=0x7f57d0006b40).stop 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 msgr2=0x7f57d406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f57c4005010 tx=0x7f57c400b040).stop 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 shutdown_connections 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f57c002ff10 0x7f57c00323d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:23.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 --2- 172.21.15.203:0/43988531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f57d40f6510 0x7f57d406b9a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:23.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 >> 172.21.15.203:0/43988531 conn(0x7f57d40f1c70 msgr2=0x7f57d40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:23.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 shutdown_connections 2024-02-19T22:18:23.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:23.379+0000 7f57db564700 1 -- 172.21.15.203:0/43988531 wait complete. 2024-02-19T22:18:23.383 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:23.204687+0000 mgr.smithi111.ftjzov (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:23.378618+0000 mon.smithi111 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.203:0/43988531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:25.064 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:25.065 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:26.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:25.204922+0000 mgr.smithi111.ftjzov (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:26.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 -- 172.21.15.203:0/300708446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 msgr2=0x7f37400f4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:26.978 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 --2- 172.21.15.203:0/300708446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f4c20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3728009fb0 tx=0x7f3728009b30).stop 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 -- 172.21.15.203:0/300708446 shutdown_connections 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 --2- 172.21.15.203:0/300708446 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f4c20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 -- 172.21.15.203:0/300708446 >> 172.21.15.203:0/300708446 conn(0x7f37400f0260 msgr2=0x7f37400f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 -- 172.21.15.203:0/300708446 shutdown_connections 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.974+0000 7f37462fb700 1 -- 172.21.15.203:0/300708446 wait complete. 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f37462fb700 1 Processor -- start 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f37462fb700 1 -- start start 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f37462fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f37462fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37400f5870 con 0x7f37400f4800 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f373ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f373ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51530/0 (socket says 172.21.15.203:51530) 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.975+0000 7f373ffff700 1 -- 172.21.15.203:0/1439983425 learned_addr learned my addr 172.21.15.203:0/1439983425 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:26.979 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f373ffff700 1 -- 172.21.15.203:0/1439983425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3728005040 con 0x7f37400f4800 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f373ffff700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3728009830 tx=0x7f372800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3728009320 con 0x7f37400f4800 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3728013380 con 0x7f37400f4800 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37400f5a10 con 0x7f37400f4800 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3728014420 con 0x7f37400f4800 2024-02-19T22:18:26.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.976+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37400f5ea0 con 0x7f37400f4800 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.977+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f372801e420 con 0x7f37400f4800 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.977+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3740059100 con 0x7f37400f4800 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.977+0000 7f373cff9700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 0x7f372c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.977+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3728040ef0 con 0x7f37400f4800 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.977+0000 7f373f7fe700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 0x7f372c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:26.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.978+0000 7f373f7fe700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 0x7f372c032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f37300096a0 tx=0x7f3730006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:26.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:26.982+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3728014aa0 con 0x7f37400f4800 2024-02-19T22:18:27.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.187+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3740045ad0 con 0x7f37400f4800 2024-02-19T22:18:27.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.188+0000 7f373cff9700 1 -- 172.21.15.203:0/1439983425 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3728004490 con 0x7f37400f4800 2024-02-19T22:18:27.190 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:27.190 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 msgr2=0x7f372c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 0x7f372c032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f37300096a0 tx=0x7f3730006b40).stop 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 msgr2=0x7f37400f73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3728009830 tx=0x7f372800b040).stop 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 shutdown_connections 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f372c02ff60 0x7f372c032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:27.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 --2- 172.21.15.203:0/1439983425 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f37400f4800 0x7f37400f73f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:27.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 >> 172.21.15.203:0/1439983425 conn(0x7f37400f0260 msgr2=0x7f37400f0cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:27.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 shutdown_connections 2024-02-19T22:18:27.193 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:27.190+0000 7f37462fb700 1 -- 172.21.15.203:0/1439983425 wait complete. 2024-02-19T22:18:27.193 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:27.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:27.189279+0000 mon.smithi111 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.203:0/1439983425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:27.205270+0000 mgr.smithi111.ftjzov (mgr.14162) 156 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:28.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:28.863 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:30.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:29.205636+0000 mgr.smithi111.ftjzov (mgr.14162) 157 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:30.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:29.875438+0000 mon.smithi111 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T22:18:30.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:29.877975+0000 mon.smithi111 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T22:18:30.603 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:29.878772+0000 mon.smithi111 (mon.0) 202 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T22:18:30.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 -- 172.21.15.203:0/1088965978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 msgr2=0x7f59540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 --2- 172.21.15.203:0/1088965978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59540f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f5948009fb0 tx=0x7f5948009b30).stop 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 -- 172.21.15.203:0/1088965978 shutdown_connections 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 --2- 172.21.15.203:0/1088965978 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59540f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 -- 172.21.15.203:0/1088965978 >> 172.21.15.203:0/1088965978 conn(0x7f59540f0b70 msgr2=0x7f59540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 -- 172.21.15.203:0/1088965978 shutdown_connections 2024-02-19T22:18:30.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.858+0000 7f59593e8700 1 -- 172.21.15.203:0/1088965978 wait complete. 2024-02-19T22:18:30.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f59593e8700 1 Processor -- start 2024-02-19T22:18:30.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f59593e8700 1 -- start start 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f59593e8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f59593e8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5954060280 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f5952ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51554/0 (socket says 172.21.15.203:51554) 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f5952ffd700 1 -- 172.21.15.203:0/332664169 learned_addr learned my addr 172.21.15.203:0/332664169 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.859+0000 7f5952ffd700 1 -- 172.21.15.203:0/332664169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5948005040 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f5952ffd700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f59480089f0 tx=0x7f594800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5948009320 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5948013380 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f594801ed80 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5954060480 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5954109c10 con 0x7f59540f5410 2024-02-19T22:18:30.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.860+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f594801e420 con 0x7f59540f5410 2024-02-19T22:18:30.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.861+0000 7f593b7fe700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 0x7f593c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:30.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.861+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5948041130 con 0x7f59540f5410 2024-02-19T22:18:30.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.861+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5954045ad0 con 0x7f59540f5410 2024-02-19T22:18:30.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.861+0000 7f59527fc700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 0x7f593c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:30.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.861+0000 7f59527fc700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 0x7f593c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f59440096a0 tx=0x7f5944006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:30.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:30.865+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59480095f0 con 0x7f59540f5410 2024-02-19T22:18:31.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.070+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59540fafa0 con 0x7f59540f5410 2024-02-19T22:18:31.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.071+0000 7f593b7fe700 1 -- 172.21.15.203:0/332664169 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59540fafa0 con 0x7f59540f5410 2024-02-19T22:18:31.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:31.073 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 msgr2=0x7f593c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 0x7f593c032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f59440096a0 tx=0x7f5944006b40).stop 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 msgr2=0x7f59541094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f59480089f0 tx=0x7f594800b040).stop 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 shutdown_connections 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f593c02ff60 0x7f593c032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 --2- 172.21.15.203:0/332664169 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f59540f5410 0x7f59541094e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 >> 172.21.15.203:0/332664169 conn(0x7f59540f0b70 msgr2=0x7f59540f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:31.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.072+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 shutdown_connections 2024-02-19T22:18:31.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:31.073+0000 7f59593e8700 1 -- 172.21.15.203:0/332664169 wait complete. 2024-02-19T22:18:31.076 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:31.534 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:18:29.874162+0000 mgr.smithi111.ftjzov (mgr.14162) 158 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi111 ... 2024-02-19T22:18:31.534 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:18:31.534 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:18:31.534 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:14:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:14:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:15:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:16:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:17:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:17:52Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:18:31.535 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('uid/gid not found') 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: uid/gid not found 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: yield (conn, connr) 2024-02-19T22:18:31.536 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: code, '\n'.join(err))) 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi111 ... 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:14:08Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:14:45Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:15:23Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:16:37Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:17:14Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:17:52Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:18:31.537 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('uid/gid not found') 2024-02-19T22:18:31.538 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: uid/gid not found 2024-02-19T22:18:31.539 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:18:29.879398+0000 mgr.smithi111.ftjzov (mgr.14162) 159 : cephadm [INF] Deploying daemon crash.smithi111 on smithi111 2024-02-19T22:18:31.539 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:31 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:31.072144+0000 mon.smithi111 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.203:0/332664169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:32.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:31.205991+0000 mgr.smithi111.ftjzov (mgr.14162) 160 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:32.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.166486+0000 mon.smithi111 (mon.0) 204 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:18:32.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:32.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.305640+0000 mon.smithi111 (mon.0) 205 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.310759+0000 mon.smithi111 (mon.0) 206 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.311205+0000 mon.smithi111 (mon.0) 207 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.311617+0000 mgr.smithi111.ftjzov (mgr.14162) 161 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:32.316704+0000 mon.smithi111 (mon.0) 208 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:18:33.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:18:32.319681+0000 mgr.smithi111.ftjzov (mgr.14162) 162 : cephadm [INF] Deploying daemon grafana.smithi111 on smithi111 2024-02-19T22:18:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:33.206400+0000 mgr.smithi111.ftjzov (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- 172.21.15.203:0/2050663116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 msgr2=0x7f02bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 --2- 172.21.15.203:0/2050663116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc0f6930 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f02b0009fb0 tx=0x7f02b0009b30).stop 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- 172.21.15.203:0/2050663116 shutdown_connections 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 --2- 172.21.15.203:0/2050663116 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc0f6930 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- 172.21.15.203:0/2050663116 >> 172.21.15.203:0/2050663116 conn(0x7f02bc0f1c70 msgr2=0x7f02bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:34.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- 172.21.15.203:0/2050663116 shutdown_connections 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- 172.21.15.203:0/2050663116 wait complete. 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 Processor -- start 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.812+0000 7f02c252f700 1 -- start start 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02c252f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02c252f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc069e20 con 0x7f02bc0f6510 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02bbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51564/0 (socket says 172.21.15.203:51564) 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02bbfff700 1 -- 172.21.15.203:0/358613142 learned_addr learned my addr 172.21.15.203:0/358613142 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02bbfff700 1 -- 172.21.15.203:0/358613142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b0005040 con 0x7f02bc0f6510 2024-02-19T22:18:34.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02bbfff700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f02b0005010 tx=0x7f02b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:34.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b0009320 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc069fc0 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f02b0013380 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02b0014e50 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.813+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc06a450 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.814+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02b0014420 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.814+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc059100 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.814+0000 7f02b8ff9700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 0x7f02a40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.814+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02b0040e30 con 0x7f02bc0f6510 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.815+0000 7f02bb7fe700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 0x7f02a40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:34.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.815+0000 7f02bb7fe700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 0x7f02a40323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f02ac0096a0 tx=0x7f02ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:34.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:34.819+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02b00095f0 con 0x7f02bc0f6510 2024-02-19T22:18:35.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.026+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02bc0fc0a0 con 0x7f02bc0f6510 2024-02-19T22:18:35.029 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.026+0000 7f02b8ff9700 1 -- 172.21.15.203:0/358613142 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f02b001db70 con 0x7f02bc0f6510 2024-02-19T22:18:35.029 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:35.029 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 msgr2=0x7f02a40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 0x7f02a40323d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f02ac0096a0 tx=0x7f02ac006b40).stop 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 msgr2=0x7f02bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f02b0005010 tx=0x7f02b000b040).stop 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 shutdown_connections 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f02a402ff10 0x7f02a40323d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:35.030 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 --2- 172.21.15.203:0/358613142 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02bc0f6510 0x7f02bc06b9a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:35.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 >> 172.21.15.203:0/358613142 conn(0x7f02bc0f1c70 msgr2=0x7f02bc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:35.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 shutdown_connections 2024-02-19T22:18:35.031 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:35.028+0000 7f02c252f700 1 -- 172.21.15.203:0/358613142 wait complete. 2024-02-19T22:18:35.031 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:35.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:35 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:35.027629+0000 mon.smithi111 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.203:0/358613142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:35.206686+0000 mgr.smithi111.ftjzov (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:36.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:36.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:37.207012+0000 mgr.smithi111.ftjzov (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:38.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 -- 172.21.15.203:0/1608496844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 msgr2=0x7f42600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 --2- 172.21.15.203:0/1608496844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42600f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f4250009fb0 tx=0x7f4250009b30).stop 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 -- 172.21.15.203:0/1608496844 shutdown_connections 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 --2- 172.21.15.203:0/1608496844 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42600f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 -- 172.21.15.203:0/1608496844 >> 172.21.15.203:0/1608496844 conn(0x7f42600f1c70 msgr2=0x7f42600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 -- 172.21.15.203:0/1608496844 shutdown_connections 2024-02-19T22:18:38.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.749+0000 7f42683ba700 1 -- 172.21.15.203:0/1608496844 wait complete. 2024-02-19T22:18:38.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.750+0000 7f42683ba700 1 Processor -- start 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.750+0000 7f42683ba700 1 -- start start 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.750+0000 7f42683ba700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.750+0000 7f42683ba700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260106cd0 con 0x7f42600f6510 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.750+0000 7f4266156700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4266156700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:35496/0 (socket says 172.21.15.203:35496) 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4266156700 1 -- 172.21.15.203:0/3803358565 learned_addr learned my addr 172.21.15.203:0/3803358565 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4266156700 1 -- 172.21.15.203:0/3803358565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250005040 con 0x7f42600f6510 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4266156700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f4250005010 tx=0x7f425000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4250009320 con 0x7f42600f6510 2024-02-19T22:18:38.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260106ed0 con 0x7f42600f6510 2024-02-19T22:18:38.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4250013380 con 0x7f42600f6510 2024-02-19T22:18:38.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4250014e50 con 0x7f42600f6510 2024-02-19T22:18:38.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.751+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260107360 con 0x7f42600f6510 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.752+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f42500095f0 con 0x7f42600f6510 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.752+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260045ad0 con 0x7f42600f6510 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.752+0000 7f4256ffd700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 0x7f424c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.752+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4250040e70 con 0x7f42600f6510 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.752+0000 7f4265955700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 0x7f424c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:38.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.753+0000 7f4265955700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 0x7f424c0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f425c0096a0 tx=0x7f425c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:38.759 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.757+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4250014420 con 0x7f42600f6510 2024-02-19T22:18:38.967 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.965+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f42601081b0 con 0x7f42600f6510 2024-02-19T22:18:38.968 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.966+0000 7f4256ffd700 1 -- 172.21.15.203:0/3803358565 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4250010230 con 0x7f42600f6510 2024-02-19T22:18:38.968 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:38.969 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:38.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 msgr2=0x7f424c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:38.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 0x7f424c0323d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f425c0096a0 tx=0x7f425c006b40).stop 2024-02-19T22:18:38.970 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 msgr2=0x7f42601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f4250005010 tx=0x7f425000b040).stop 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 shutdown_connections 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f424c02ff10 0x7f424c0323d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 --2- 172.21.15.203:0/3803358565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f42600f6510 0x7f42601067e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 >> 172.21.15.203:0/3803358565 conn(0x7f42600f1c70 msgr2=0x7f42600f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 shutdown_connections 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:38.968+0000 7f42683ba700 1 -- 172.21.15.203:0/3803358565 wait complete. 2024-02-19T22:18:38.971 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:39.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:39 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:38.967486+0000 mon.smithi111 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.203:0/3803358565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:40.659 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:40.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:40.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:39.207324+0000 mgr.smithi111.ftjzov (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:42.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:41.207561+0000 mgr.smithi111.ftjzov (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/3021454822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 msgr2=0x7fcc940f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/3021454822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc940f7d00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fcc84009fb0 tx=0x7fcc84009b30).stop 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/3021454822 shutdown_connections 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/3021454822 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc940f7d00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/3021454822 >> 172.21.15.203:0/3021454822 conn(0x7fcc940f0120 msgr2=0x7fcc940f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/3021454822 shutdown_connections 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/3021454822 wait complete. 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 Processor -- start 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.577+0000 7fcc9b3bd700 1 -- start start 2024-02-19T22:18:42.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9b3bd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9b3bd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94106aa0 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9a3bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9a3bb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:35508/0 (socket says 172.21.15.203:35508) 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9a3bb700 1 -- 172.21.15.203:0/542509097 learned_addr learned my addr 172.21.15.203:0/542509097 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9a3bb700 1 -- 172.21.15.203:0/542509097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc84005040 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.578+0000 7fcc9a3bb700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fcc84009830 tx=0x7fcc8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc84009320 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcc84013380 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc94106ca0 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc84014420 con 0x7fcc940f78e0 2024-02-19T22:18:42.582 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc94107130 con 0x7fcc940f78e0 2024-02-19T22:18:42.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc8401e420 con 0x7fcc940f78e0 2024-02-19T22:18:42.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 0x7fcc80032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:42.583 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc94045ad0 con 0x7fcc940f78e0 2024-02-19T22:18:42.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.579+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc84040eb0 con 0x7fcc940f78e0 2024-02-19T22:18:42.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.580+0000 7fcc99bba700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 0x7fcc80032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:42.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.580+0000 7fcc99bba700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 0x7fcc80032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcc900096a0 tx=0x7fcc90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:42.586 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.584+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc8401cb10 con 0x7fcc940f78e0 2024-02-19T22:18:42.791 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.789+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc940fbe20 con 0x7fcc940f78e0 2024-02-19T22:18:42.791 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.789+0000 7fcc8affd700 1 -- 172.21.15.203:0/542509097 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc8401cb10 con 0x7fcc940f78e0 2024-02-19T22:18:42.792 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:42.792 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:42.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 msgr2=0x7fcc80032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:42.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 0x7fcc80032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fcc900096a0 tx=0x7fcc90006b40).stop 2024-02-19T22:18:42.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 msgr2=0x7fcc941065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:42.793 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fcc84009830 tx=0x7fcc8400b040).stop 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 shutdown_connections 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fcc8002ffb0 0x7fcc80032470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 --2- 172.21.15.203:0/542509097 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fcc940f78e0 0x7fcc941065b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 >> 172.21.15.203:0/542509097 conn(0x7fcc940f0120 msgr2=0x7fcc940f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 shutdown_connections 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:42.791+0000 7fcc9b3bd700 1 -- 172.21.15.203:0/542509097 wait complete. 2024-02-19T22:18:42.794 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:43.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:42.790529+0000 mon.smithi111 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.203:0/542509097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:44.525 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:44.525 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:43.207784+0000 mgr.smithi111.ftjzov (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:46.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/275673107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 msgr2=0x7fd8d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:46.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/275673107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d40f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fd8c8009fb0 tx=0x7fd8c8009b30).stop 2024-02-19T22:18:46.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/275673107 shutdown_connections 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/275673107 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d40f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/275673107 >> 172.21.15.203:0/275673107 conn(0x7fd8d40f0b70 msgr2=0x7fd8d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/275673107 shutdown_connections 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.324+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/275673107 wait complete. 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8dbf4e700 1 Processor -- start 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8dbf4e700 1 -- start start 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8dbf4e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8dbf4e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d4060220 con 0x7fd8d40f5410 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8d9cea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8d9cea700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58802/0 (socket says 172.21.15.203:58802) 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8d9cea700 1 -- 172.21.15.203:0/2222200991 learned_addr learned my addr 172.21.15.203:0/2222200991 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:46.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.325+0000 7fd8d9cea700 1 -- 172.21.15.203:0/2222200991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c8005040 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8d9cea700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd8c8005010 tx=0x7fd8c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c8009320 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8c8013380 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c8014e50 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d4060420 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.326+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d4109be0 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8c80095f0 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d4059100 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8c67fc700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 0x7fd8c00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8c8040e60 con 0x7fd8d40f5410 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8d94e9700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 0x7fd8c00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:46.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.327+0000 7fd8d94e9700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 0x7fd8c00323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd8d00096a0 tx=0x7fd8d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:46.340 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.331+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8c8014420 con 0x7fd8d40f5410 2024-02-19T22:18:46.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.535+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8d4045ad0 con 0x7fd8d40f5410 2024-02-19T22:18:46.537 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.535+0000 7fd8c67fc700 1 -- 172.21.15.203:0/2222200991 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd8c801d980 con 0x7fd8d40f5410 2024-02-19T22:18:46.538 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:46.538 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 msgr2=0x7fd8c00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 0x7fd8c00323d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fd8d00096a0 tx=0x7fd8d0006b40).stop 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 msgr2=0x7fd8d4109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fd8c8005010 tx=0x7fd8c800b040).stop 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 shutdown_connections 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd8c002ff10 0x7fd8c00323d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 --2- 172.21.15.203:0/2222200991 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd8d40f5410 0x7fd8d4109530 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:46.539 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 >> 172.21.15.203:0/2222200991 conn(0x7fd8d40f0b70 msgr2=0x7fd8d40f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:46.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 shutdown_connections 2024-02-19T22:18:46.540 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:46.537+0000 7fd8dbf4e700 1 -- 172.21.15.203:0/2222200991 wait complete. 2024-02-19T22:18:46.540 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:45.208025+0000 mgr.smithi111.ftjzov (mgr.14162) 169 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:47.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:47 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:46.536553+0000 mon.smithi111 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.203:0/2222200991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:48.127 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:48.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:47.208283+0000 mgr.smithi111.ftjzov (mgr.14162) 170 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 -- 172.21.15.203:0/2330575236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 msgr2=0x7fef000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 --2- 172.21.15.203:0/2330575236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef000f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7feef4009fb0 tx=0x7feef4009b30).stop 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 -- 172.21.15.203:0/2330575236 shutdown_connections 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 --2- 172.21.15.203:0/2330575236 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef000f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 -- 172.21.15.203:0/2330575236 >> 172.21.15.203:0/2330575236 conn(0x7fef000f1c70 msgr2=0x7fef000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 -- 172.21.15.203:0/2330575236 shutdown_connections 2024-02-19T22:18:50.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.055+0000 7fef06e4f700 1 -- 172.21.15.203:0/2330575236 wait complete. 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.056+0000 7fef06e4f700 1 Processor -- start 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.056+0000 7fef06e4f700 1 -- start start 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.056+0000 7fef06e4f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.056+0000 7fef06e4f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00106cd0 con 0x7fef000f6510 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef04beb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef04beb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58806/0 (socket says 172.21.15.203:58806) 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef04beb700 1 -- 172.21.15.203:0/3310329685 learned_addr learned my addr 172.21.15.203:0/3310329685 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef04beb700 1 -- 172.21.15.203:0/3310329685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef4005040 con 0x7fef000f6510 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef04beb700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7feef4005010 tx=0x7feef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:50.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef4009320 con 0x7fef000f6510 2024-02-19T22:18:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00106ed0 con 0x7fef000f6510 2024-02-19T22:18:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feef4013380 con 0x7fef000f6510 2024-02-19T22:18:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef4014e50 con 0x7fef000f6510 2024-02-19T22:18:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.057+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef00107360 con 0x7fef000f6510 2024-02-19T22:18:50.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.058+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feef40263a0 con 0x7fef000f6510 2024-02-19T22:18:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.058+0000 7feefd7fa700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 0x7feef00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.058+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feef4041f10 con 0x7fef000f6510 2024-02-19T22:18:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.058+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00059100 con 0x7fef000f6510 2024-02-19T22:18:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.058+0000 7feefffff700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 0x7feef00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:50.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.059+0000 7feefffff700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 0x7feef00323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7feeec0096a0 tx=0x7feeec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:50.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.063+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feef4012920 con 0x7fef000f6510 2024-02-19T22:18:50.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.267+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef000fc0a0 con 0x7fef000f6510 2024-02-19T22:18:50.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.267+0000 7feefd7fa700 1 -- 172.21.15.203:0/3310329685 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feef402b020 con 0x7fef000f6510 2024-02-19T22:18:50.270 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:50.270 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 msgr2=0x7feef00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 0x7feef00323d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7feeec0096a0 tx=0x7feeec006b40).stop 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 msgr2=0x7fef001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7feef4005010 tx=0x7feef400b040).stop 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 shutdown_connections 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7feef002ff10 0x7feef00323d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 --2- 172.21.15.203:0/3310329685 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fef000f6510 0x7fef001067e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 >> 172.21.15.203:0/3310329685 conn(0x7fef000f1c70 msgr2=0x7fef000f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 shutdown_connections 2024-02-19T22:18:50.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:50.269+0000 7fef06e4f700 1 -- 172.21.15.203:0/3310329685 wait complete. 2024-02-19T22:18:50.273 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:50.327 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:49.208516+0000 mgr.smithi111.ftjzov (mgr.14162) 171 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:51.612 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:50.268699+0000 mon.smithi111 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.203:0/3310329685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:51.983 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:51.984 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:52.599 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:51.208820+0000 mgr.smithi111.ftjzov (mgr.14162) 172 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:54.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 -- 172.21.15.203:0/771829876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 msgr2=0x7fdb6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 --2- 172.21.15.203:0/771829876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c0f6930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fdb64009fb0 tx=0x7fdb64009b30).stop 2024-02-19T22:18:54.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 -- 172.21.15.203:0/771829876 shutdown_connections 2024-02-19T22:18:54.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 --2- 172.21.15.203:0/771829876 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c0f6930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 -- 172.21.15.203:0/771829876 >> 172.21.15.203:0/771829876 conn(0x7fdb6c0f1c70 msgr2=0x7fdb6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 -- 172.21.15.203:0/771829876 shutdown_connections 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.908+0000 7fdb74037700 1 -- 172.21.15.203:0/771829876 wait complete. 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb74037700 1 Processor -- start 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb74037700 1 -- start start 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb74037700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb74037700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c106cd0 con 0x7fdb6c0f6510 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb71dd3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb71dd3700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:58820/0 (socket says 172.21.15.203:58820) 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.909+0000 7fdb71dd3700 1 -- 172.21.15.203:0/1863836659 learned_addr learned my addr 172.21.15.203:0/1863836659 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb71dd3700 1 -- 172.21.15.203:0/1863836659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64005040 con 0x7fdb6c0f6510 2024-02-19T22:18:54.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb71dd3700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdb640089f0 tx=0x7fdb6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb64009320 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb64013380 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c106ed0 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb6401ed80 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.910+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c107360 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.911+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb6401e420 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.911+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c045ad0 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.911+0000 7fdb6a7fc700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 0x7fdb540324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.911+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb64041190 con 0x7fdb6c0f6510 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.911+0000 7fdb715d2700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 0x7fdb540324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.912+0000 7fdb715d2700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 0x7fdb540324c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdb600096a0 tx=0x7fdb60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:54.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:53.916+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb6401ccf0 con 0x7fdb6c0f6510 2024-02-19T22:18:54.125 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.124+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb6c108210 con 0x7fdb6c0f6510 2024-02-19T22:18:54.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.125+0000 7fdb6a7fc700 1 -- 172.21.15.203:0/1863836659 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb6401cb10 con 0x7fdb6c0f6510 2024-02-19T22:18:54.126 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:54.127 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 msgr2=0x7fdb540324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 0x7fdb540324c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fdb600096a0 tx=0x7fdb60006b40).stop 2024-02-19T22:18:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 msgr2=0x7fdb6c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fdb640089f0 tx=0x7fdb6400b040).stop 2024-02-19T22:18:54.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 shutdown_connections 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdb54030000 0x7fdb540324c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 --2- 172.21.15.203:0/1863836659 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdb6c0f6510 0x7fdb6c1067e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 >> 172.21.15.203:0/1863836659 conn(0x7fdb6c0f1c70 msgr2=0x7fdb6c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 shutdown_connections 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:54.127+0000 7fdb74037700 1 -- 172.21.15.203:0/1863836659 wait complete. 2024-02-19T22:18:54.129 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:54.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:52.743157+0000 mon.smithi111 (mon.0) 214 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:18:54.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:18:52.744710+0000 mgr.smithi111.ftjzov (mgr.14162) 173 : cephadm [INF] Deploying daemon node-exporter.smithi111 on smithi111 2024-02-19T22:18:54.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:53.209054+0000 mgr.smithi111.ftjzov (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:55.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:54.125322+0000 mon.smithi111 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.203:0/1863836659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:55.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:55.798 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:18:56.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:55.209267+0000 mgr.smithi111.ftjzov (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:57.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.568+0000 7f70385d4700 1 -- 172.21.15.203:0/1951053258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 msgr2=0x7f70300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:57.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.568+0000 7f70385d4700 1 --2- 172.21.15.203:0/1951053258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f70300f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f7020009fb0 tx=0x7f7020009b30).stop 2024-02-19T22:18:57.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- 172.21.15.203:0/1951053258 shutdown_connections 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 --2- 172.21.15.203:0/1951053258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f70300f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- 172.21.15.203:0/1951053258 >> 172.21.15.203:0/1951053258 conn(0x7f70300f0b70 msgr2=0x7f70300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- 172.21.15.203:0/1951053258 shutdown_connections 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- 172.21.15.203:0/1951053258 wait complete. 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 Processor -- start 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- start start 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f70385d4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7030060220 con 0x7f70300f5410 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.569+0000 7f7036370700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7036370700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33414/0 (socket says 172.21.15.203:33414) 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7036370700 1 -- 172.21.15.203:0/208579410 learned_addr learned my addr 172.21.15.203:0/208579410 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7036370700 1 -- 172.21.15.203:0/208579410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020005040 con 0x7f70300f5410 2024-02-19T22:18:57.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7036370700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7020005010 tx=0x7f702000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:57.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7020009320 con 0x7f70300f5410 2024-02-19T22:18:57.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7030060420 con 0x7f70300f5410 2024-02-19T22:18:57.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7020013380 con 0x7f70300f5410 2024-02-19T22:18:57.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7020014e50 con 0x7f70300f5410 2024-02-19T22:18:57.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.570+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7030109bb0 con 0x7f70300f5410 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.571+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70200095f0 con 0x7f70300f5410 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.571+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7030059100 con 0x7f70300f5410 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.571+0000 7f7026ffd700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 0x7f701c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.571+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7020040e90 con 0x7f70300f5410 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.571+0000 7f7035b6f700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 0x7f701c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:18:57.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.572+0000 7f7035b6f700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 0x7f701c0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f702c0096a0 tx=0x7f702c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:18:57.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.576+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7020014420 con 0x7f70300f5410 2024-02-19T22:18:57.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.781+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7030045ad0 con 0x7f70300f5410 2024-02-19T22:18:57.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.782+0000 7f7026ffd700 1 -- 172.21.15.203:0/208579410 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7020010230 con 0x7f70300f5410 2024-02-19T22:18:57.783 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:18:57.783 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:18:57.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 msgr2=0x7f701c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 0x7f701c0323d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f702c0096a0 tx=0x7f702c006b40).stop 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 msgr2=0x7f7030109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f7020005010 tx=0x7f702000b040).stop 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 shutdown_connections 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f701c02ff10 0x7f701c0323d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 --2- 172.21.15.203:0/208579410 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f70300f5410 0x7f7030109500 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 >> 172.21.15.203:0/208579410 conn(0x7f70300f0b70 msgr2=0x7f70300f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 shutdown_connections 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:18:57.783+0000 7f70385d4700 1 -- 172.21.15.203:0/208579410 wait complete. 2024-02-19T22:18:57.785 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:18:58.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:57.209525+0000 mgr.smithi111.ftjzov (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:18:58.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:18:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:18:57.782008+0000 mon.smithi111 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.203:0/208579410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:18:59.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:18:59.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:18:59.209867+0000 mgr.smithi111.ftjzov (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 -- 172.21.15.203:0/429691646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 msgr2=0x7f595c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 --2- 172.21.15.203:0/429691646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c0f90f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5950009fb0 tx=0x7f5950009b30).stop 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 -- 172.21.15.203:0/429691646 shutdown_connections 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 --2- 172.21.15.203:0/429691646 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c0f90f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 -- 172.21.15.203:0/429691646 >> 172.21.15.203:0/429691646 conn(0x7f595c0f0b70 msgr2=0x7f595c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 -- 172.21.15.203:0/429691646 shutdown_connections 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.452+0000 7f5961306700 1 -- 172.21.15.203:0/429691646 wait complete. 2024-02-19T22:19:01.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f5961306700 1 Processor -- start 2024-02-19T22:19:01.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f5961306700 1 -- start start 2024-02-19T22:19:01.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f5961306700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:01.456 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f5961306700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c106c70 con 0x7f595c060280 2024-02-19T22:19:01.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f595affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:01.457 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f595affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33424/0 (socket says 172.21.15.203:33424) 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f595affd700 1 -- 172.21.15.203:0/2108623538 learned_addr learned my addr 172.21.15.203:0/2108623538 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.453+0000 7f595affd700 1 -- 172.21.15.203:0/2108623538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950005040 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f595affd700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f59500089f0 tx=0x7f595000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5950009320 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c106e70 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5950013380 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5950014e20 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.454+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c107300 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59500095f0 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c045ad0 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f59437fe700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 0x7f59440323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5950040e90 con 0x7f595c060280 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f595a7fc700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 0x7f59440323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:01.458 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.455+0000 7f595a7fc700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 0x7f59440323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f594c0096a0 tx=0x7f594c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:01.462 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.460+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59500268c0 con 0x7f595c060280 2024-02-19T22:19:01.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.667+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f595c1081b0 con 0x7f595c060280 2024-02-19T22:19:01.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.668+0000 7f59437fe700 1 -- 172.21.15.203:0/2108623538 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f595001d980 con 0x7f595c060280 2024-02-19T22:19:01.670 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:01.670 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 msgr2=0x7f59440323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 0x7f59440323d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f594c0096a0 tx=0x7f594c006b40).stop 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 msgr2=0x7f595c106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f59500089f0 tx=0x7f595000b040).stop 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 shutdown_connections 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f594402ff10 0x7f59440323d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 --2- 172.21.15.203:0/2108623538 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f595c060280 0x7f595c106780 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:01.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.669+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 >> 172.21.15.203:0/2108623538 conn(0x7f595c0f0b70 msgr2=0x7f595c0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:01.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.670+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 shutdown_connections 2024-02-19T22:19:01.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:01.670+0000 7f5961306700 1 -- 172.21.15.203:0/2108623538 wait complete. 2024-02-19T22:19:01.673 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:01.210184+0000 mgr.smithi111.ftjzov (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:01.668898+0000 mon.smithi111 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.203:0/2108623538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:03.459 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:03.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:03.210453+0000 mgr.smithi111.ftjzov (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.332+0000 7fb3f140f700 1 -- 172.21.15.203:0/2286548643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 msgr2=0x7fb3ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.332+0000 7fb3f140f700 1 --2- 172.21.15.203:0/2286548643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec0f55c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fb3dc009fb0 tx=0x7fb3dc009b30).stop 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 -- 172.21.15.203:0/2286548643 shutdown_connections 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 --2- 172.21.15.203:0/2286548643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec0f55c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 -- 172.21.15.203:0/2286548643 >> 172.21.15.203:0/2286548643 conn(0x7fb3ec0f09a0 msgr2=0x7fb3ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 -- 172.21.15.203:0/2286548643 shutdown_connections 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 -- 172.21.15.203:0/2286548643 wait complete. 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 Processor -- start 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.333+0000 7fb3f140f700 1 -- start start 2024-02-19T22:19:05.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3f140f700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:05.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3f140f700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1059c0 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3ebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3ebfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41800/0 (socket says 172.21.15.203:41800) 2024-02-19T22:19:05.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3ebfff700 1 -- 172.21.15.203:0/1138502408 learned_addr learned my addr 172.21.15.203:0/1138502408 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3ebfff700 1 -- 172.21.15.203:0/1138502408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc005040 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3ebfff700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb3dc0089f0 tx=0x7fb3dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3dc009320 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec105bc0 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.334+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3dc013380 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.335+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3dc014e20 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.335+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec106050 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.335+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3dc0095f0 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.335+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec045ad0 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.335+0000 7fb3e8ff9700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.336+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3dc040f50 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.336+0000 7fb3eb7fe700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:05.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.336+0000 7fb3eb7fe700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 0x7fb3d4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb3e00096a0 tx=0x7fb3e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:05.342 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.340+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3dc014420 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.546+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3ec106f00 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.547+0000 7fb3e8ff9700 1 -- 172.21.15.203:0/1138502408 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb3dc01d980 con 0x7fb3ec0f51a0 2024-02-19T22:19:05.549 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:05.549 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:05.550 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 msgr2=0x7fb3d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 0x7fb3d4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb3e00096a0 tx=0x7fb3e0006b40).stop 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 msgr2=0x7fb3ec1054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb3dc0089f0 tx=0x7fb3dc00b040).stop 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 shutdown_connections 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb3d402ff60 0x7fb3d4032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 --2- 172.21.15.203:0/1138502408 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb3ec0f51a0 0x7fb3ec1054d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 >> 172.21.15.203:0/1138502408 conn(0x7fb3ec0f09a0 msgr2=0x7fb3ec0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 shutdown_connections 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:05.548+0000 7fb3f140f700 1 -- 172.21.15.203:0/1138502408 wait complete. 2024-02-19T22:19:05.551 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:05.210759+0000 mgr.smithi111.ftjzov (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:05.548051+0000 mon.smithi111 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.203:0/1138502408' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:07.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:07.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:07.211095+0000 mgr.smithi111.ftjzov (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:09.310 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.306+0000 7fc7506f8700 1 -- 172.21.15.203:0/3453044545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 msgr2=0x7fc7480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.306+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3453044545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7480f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc738009fb0 tx=0x7fc738009b30).stop 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- 172.21.15.203:0/3453044545 shutdown_connections 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3453044545 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7480f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- 172.21.15.203:0/3453044545 >> 172.21.15.203:0/3453044545 conn(0x7fc7480f0b70 msgr2=0x7fc7480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- 172.21.15.203:0/3453044545 shutdown_connections 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- 172.21.15.203:0/3453044545 wait complete. 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 Processor -- start 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- start start 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.307+0000 7fc7506f8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748108ed0 con 0x7fc7480f5410 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc74e494700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:09.311 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc74e494700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41814/0 (socket says 172.21.15.203:41814) 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc74e494700 1 -- 172.21.15.203:0/3531423221 learned_addr learned my addr 172.21.15.203:0/3531423221 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc74e494700 1 -- 172.21.15.203:0/3531423221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc738005040 con 0x7fc7480f5410 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc74e494700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc7380089f0 tx=0x7fc73800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc738009320 con 0x7fc7480f5410 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc738013380 con 0x7fc7480f5410 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7481090d0 con 0x7fc7480f5410 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc73801ed80 con 0x7fc7480f5410 2024-02-19T22:19:09.312 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.308+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc748109560 con 0x7fc7480f5410 2024-02-19T22:19:09.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.309+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc73801e420 con 0x7fc7480f5410 2024-02-19T22:19:09.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.309+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc748045ad0 con 0x7fc7480f5410 2024-02-19T22:19:09.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.309+0000 7fc73effd700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:09.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.309+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc738041110 con 0x7fc7480f5410 2024-02-19T22:19:09.313 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.309+0000 7fc74dc93700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:09.314 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.310+0000 7fc74dc93700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 0x7fc734032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc7440096a0 tx=0x7fc744006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:09.316 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.314+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc73801cb10 con 0x7fc7480f5410 2024-02-19T22:19:09.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.519+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc74810a410 con 0x7fc7480f5410 2024-02-19T22:19:09.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.519+0000 7fc73effd700 1 -- 172.21.15.203:0/3531423221 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc738004490 con 0x7fc7480f5410 2024-02-19T22:19:09.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:09.522 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 msgr2=0x7fc734032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 0x7fc734032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fc7440096a0 tx=0x7fc744006b40).stop 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 msgr2=0x7fc7481089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc7380089f0 tx=0x7fc73800b040).stop 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 shutdown_connections 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc73402ff60 0x7fc734032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:09.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 --2- 172.21.15.203:0/3531423221 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7480f5410 0x7fc7481089e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:09.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 >> 172.21.15.203:0/3531423221 conn(0x7fc7480f0b70 msgr2=0x7fc74805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:09.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 shutdown_connections 2024-02-19T22:19:09.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:09.521+0000 7fc7506f8700 1 -- 172.21.15.203:0/3531423221 wait complete. 2024-02-19T22:19:09.524 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:09.211471+0000 mgr.smithi111.ftjzov (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:10.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:09.520590+0000 mon.smithi111 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.203:0/3531423221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:11.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:11.288 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:11.211764+0000 mgr.smithi111.ftjzov (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.150+0000 7ff294606700 1 -- 172.21.15.203:0/1195153260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 msgr2=0x7ff28c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.150+0000 7ff294606700 1 --2- 172.21.15.203:0/1195153260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c0f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7ff288009fb0 tx=0x7ff288009b30).stop 2024-02-19T22:19:13.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.150+0000 7ff294606700 1 -- 172.21.15.203:0/1195153260 shutdown_connections 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.150+0000 7ff294606700 1 --2- 172.21.15.203:0/1195153260 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c0f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 -- 172.21.15.203:0/1195153260 >> 172.21.15.203:0/1195153260 conn(0x7ff28c0f0b70 msgr2=0x7ff28c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 -- 172.21.15.203:0/1195153260 shutdown_connections 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 -- 172.21.15.203:0/1195153260 wait complete. 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 Processor -- start 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 -- start start 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff294606700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c108ef0 con 0x7ff28c0f5410 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff2923a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff2923a2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41830/0 (socket says 172.21.15.203:41830) 2024-02-19T22:19:13.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.151+0000 7ff2923a2700 1 -- 172.21.15.203:0/680146100 learned_addr learned my addr 172.21.15.203:0/680146100 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff2923a2700 1 -- 172.21.15.203:0/680146100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff288005040 con 0x7ff28c0f5410 2024-02-19T22:19:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff2923a2700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff288005010 tx=0x7ff28800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff288009320 con 0x7ff28c0f5410 2024-02-19T22:19:13.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c1090f0 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff288013380 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff288014e50 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.152+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c109580 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2880095f0 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c045ad0 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff282ffd700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 0x7ff2780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff288040e90 con 0x7ff28c0f5410 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff291ba1700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 0x7ff2780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:13.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.153+0000 7ff291ba1700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 0x7ff2780323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff27c0096a0 tx=0x7ff27c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:13.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.157+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff288014420 con 0x7ff28c0f5410 2024-02-19T22:19:13.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.362+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff28c10a3c0 con 0x7ff28c0f5410 2024-02-19T22:19:13.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.362+0000 7ff282ffd700 1 -- 172.21.15.203:0/680146100 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff288010230 con 0x7ff28c0f5410 2024-02-19T22:19:13.365 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:13.365 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 msgr2=0x7ff2780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 0x7ff2780323d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7ff27c0096a0 tx=0x7ff27c006b40).stop 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 msgr2=0x7ff28c108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7ff288005010 tx=0x7ff28800b040).stop 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 shutdown_connections 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff27802ff10 0x7ff2780323d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 --2- 172.21.15.203:0/680146100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff28c0f5410 0x7ff28c108a00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:13.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 >> 172.21.15.203:0/680146100 conn(0x7ff28c0f0b70 msgr2=0x7ff28c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:13.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 shutdown_connections 2024-02-19T22:19:13.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:13.364+0000 7ff294606700 1 -- 172.21.15.203:0/680146100 wait complete. 2024-02-19T22:19:13.368 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:13.212083+0000 mgr.smithi111.ftjzov (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:14.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:13.363657+0000 mon.smithi111 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.203:0/680146100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:14.955 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:14.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:15.212466+0000 mgr.smithi111.ftjzov (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:16.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/3932517521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 msgr2=0x7f6a980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/3932517521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a980f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f6a88009fb0 tx=0x7f6a88009b30).stop 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/3932517521 shutdown_connections 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/3932517521 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a980f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/3932517521 >> 172.21.15.203:0/3932517521 conn(0x7f6a980f0b70 msgr2=0x7f6a980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/3932517521 shutdown_connections 2024-02-19T22:19:16.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.984+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/3932517521 wait complete. 2024-02-19T22:19:16.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a9e6a5700 1 Processor -- start 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a9e6a5700 1 -- start start 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a9e6a5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a9e6a5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a98105bd0 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a97fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a97fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43496/0 (socket says 172.21.15.203:43496) 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.985+0000 7f6a97fff700 1 -- 172.21.15.203:0/340888364 learned_addr learned my addr 172.21.15.203:0/340888364 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a97fff700 1 -- 172.21.15.203:0/340888364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88005040 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a97fff700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6a88005010 tx=0x7f6a8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a88009320 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a98105dd0 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6a88013380 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a88014e50 con 0x7f6a980f5410 2024-02-19T22:19:16.989 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.986+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a98106260 con 0x7f6a980f5410 2024-02-19T22:19:16.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.987+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a880095f0 con 0x7f6a980f5410 2024-02-19T22:19:16.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.987+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a98045ad0 con 0x7f6a980f5410 2024-02-19T22:19:16.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.987+0000 7f6a94ff9700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:16.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.987+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a88042c70 con 0x7f6a980f5410 2024-02-19T22:19:16.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.988+0000 7f6a977fe700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:16.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.988+0000 7f6a977fe700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 0x7f6a800323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:16.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:16.992+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a88014420 con 0x7f6a980f5410 2024-02-19T22:19:17.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.198+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a98107110 con 0x7f6a980f5410 2024-02-19T22:19:17.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.199+0000 7f6a94ff9700 1 -- 172.21.15.203:0/340888364 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a88011290 con 0x7f6a980f5410 2024-02-19T22:19:17.202 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:17.202 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 msgr2=0x7f6a800323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 0x7f6a800323d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6a8c0096a0 tx=0x7f6a8c006b40).stop 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 msgr2=0x7f6a981056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6a88005010 tx=0x7f6a8800b040).stop 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 shutdown_connections 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6a8002ff10 0x7f6a800323d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 --2- 172.21.15.203:0/340888364 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6a980f5410 0x7f6a981056e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 >> 172.21.15.203:0/340888364 conn(0x7f6a980f0b70 msgr2=0x7f6a980f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 shutdown_connections 2024-02-19T22:19:17.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:17.201+0000 7f6a9e6a5700 1 -- 172.21.15.203:0/340888364 wait complete. 2024-02-19T22:19:17.205 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:17.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:17.200518+0000 mon.smithi111 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.203:0/340888364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:17.212766+0000 mgr.smithi111.ftjzov (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:17.301167+0000 mon.smithi111 (mon.0) 222 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:19:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:17.303201+0000 mon.smithi111 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:19:18.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:18.862 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:19.213114+0000 mgr.smithi111.ftjzov (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:20.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- 172.21.15.203:0/96270057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 msgr2=0x7f8db80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 --2- 172.21.15.203:0/96270057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db80f55c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f8da8009fb0 tx=0x7f8da8009b30).stop 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- 172.21.15.203:0/96270057 shutdown_connections 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 --2- 172.21.15.203:0/96270057 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db80f55c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- 172.21.15.203:0/96270057 >> 172.21.15.203:0/96270057 conn(0x7f8db80f09a0 msgr2=0x7f8db80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- 172.21.15.203:0/96270057 shutdown_connections 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- 172.21.15.203:0/96270057 wait complete. 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 Processor -- start 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- start start 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:20.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.879+0000 7f8db759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8db8060220 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43520/0 (socket says 172.21.15.203:43520) 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db659c700 1 -- 172.21.15.203:0/1940438924 learned_addr learned my addr 172.21.15.203:0/1940438924 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db659c700 1 -- 172.21.15.203:0/1940438924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da8005040 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db659c700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8da8009830 tx=0x7f8da800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da8009320 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8db8060420 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8da8013380 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8da8014420 con 0x7f8db80f51a0 2024-02-19T22:19:20.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.880+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8db81099a0 con 0x7f8db80f51a0 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.881+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8da801e420 con 0x7f8db80f51a0 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.881+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8db8059100 con 0x7f8db80f51a0 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.881+0000 7f8da6ffd700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.881+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8da8040e30 con 0x7f8db80f51a0 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.882+0000 7f8db5d9b700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:20.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.882+0000 7f8db5d9b700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 0x7f8da0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8dac0096a0 tx=0x7f8dac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:20.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:20.886+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8da801cb10 con 0x7f8db80f51a0 2024-02-19T22:19:21.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.091+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8db8045ad0 con 0x7f8db80f51a0 2024-02-19T22:19:21.094 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.092+0000 7f8da6ffd700 1 -- 172.21.15.203:0/1940438924 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8da8004490 con 0x7f8db80f51a0 2024-02-19T22:19:21.094 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:21.094 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 msgr2=0x7f8da0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 0x7f8da0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f8dac0096a0 tx=0x7f8dac006b40).stop 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 msgr2=0x7f8db81092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f8da8009830 tx=0x7f8da800b040).stop 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 shutdown_connections 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8da002ff60 0x7f8da0032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 --2- 172.21.15.203:0/1940438924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8db80f51a0 0x7f8db81092f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:21.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 >> 172.21.15.203:0/1940438924 conn(0x7f8db80f09a0 msgr2=0x7f8db80f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:21.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 shutdown_connections 2024-02-19T22:19:21.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:21.093+0000 7f8db759e700 1 -- 172.21.15.203:0/1940438924 wait complete. 2024-02-19T22:19:21.097 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:21.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:21 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:21.093010+0000 mon.smithi111 (mon.0) 224 : audit [DBG] from='client.? 172.21.15.203:0/1940438924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:22.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:21.213450+0000 mgr.smithi111.ftjzov (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:22.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:22.773 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:24.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:23.213737+0000 mgr.smithi111.ftjzov (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:24.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 -- 172.21.15.203:0/2683687023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 msgr2=0x7efc940f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:24.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 --2- 172.21.15.203:0/2683687023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc940f90f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7efc84009fb0 tx=0x7efc84009b30).stop 2024-02-19T22:19:24.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 -- 172.21.15.203:0/2683687023 shutdown_connections 2024-02-19T22:19:24.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 --2- 172.21.15.203:0/2683687023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc940f90f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:24.980 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 -- 172.21.15.203:0/2683687023 >> 172.21.15.203:0/2683687023 conn(0x7efc940f0b70 msgr2=0x7efc940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 -- 172.21.15.203:0/2683687023 shutdown_connections 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.977+0000 7efc98933700 1 -- 172.21.15.203:0/2683687023 wait complete. 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc98933700 1 Processor -- start 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc98933700 1 -- start start 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc98933700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:24.981 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc98933700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94106c70 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc9259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc9259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43544/0 (socket says 172.21.15.203:43544) 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.978+0000 7efc9259c700 1 -- 172.21.15.203:0/2948250246 learned_addr learned my addr 172.21.15.203:0/2948250246 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc9259c700 1 -- 172.21.15.203:0/2948250246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84005040 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc9259c700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7efc840089f0 tx=0x7efc8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc84009320 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc94106e70 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc84013380 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc84014e20 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.979+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc94107300 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.980+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efc84014420 con 0x7efc94060280 2024-02-19T22:19:24.983 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.980+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc94045ad0 con 0x7efc94060280 2024-02-19T22:19:24.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.980+0000 7efc82ffd700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 0x7efc7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:24.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.980+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efc84040fc0 con 0x7efc94060280 2024-02-19T22:19:24.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.980+0000 7efc91d9b700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 0x7efc7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:24.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.981+0000 7efc91d9b700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 0x7efc7c0323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efc880096a0 tx=0x7efc88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:24.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:24.985+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc840095f0 con 0x7efc94060280 2024-02-19T22:19:25.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.192+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc941081b0 con 0x7efc94060280 2024-02-19T22:19:25.195 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.193+0000 7efc82ffd700 1 -- 172.21.15.203:0/2948250246 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efc8401db70 con 0x7efc94060280 2024-02-19T22:19:25.196 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:25.196 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 msgr2=0x7efc7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 0x7efc7c0323d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7efc880096a0 tx=0x7efc88006b40).stop 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 msgr2=0x7efc94106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7efc840089f0 tx=0x7efc8400b040).stop 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 shutdown_connections 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efc7c02ff10 0x7efc7c0323d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 --2- 172.21.15.203:0/2948250246 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efc94060280 0x7efc94106780 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 >> 172.21.15.203:0/2948250246 conn(0x7efc940f0b70 msgr2=0x7efc940f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 shutdown_connections 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:25.195+0000 7efc98933700 1 -- 172.21.15.203:0/2948250246 wait complete. 2024-02-19T22:19:25.198 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:25.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:25 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:25.194570+0000 mon.smithi111 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.203:0/2948250246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:26.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:25.213988+0000 mgr.smithi111.ftjzov (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:26.882 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:26.883 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:27.214265+0000 mgr.smithi111.ftjzov (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 -- 172.21.15.203:0/3124846303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 msgr2=0x7fa7a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 --2- 172.21.15.203:0/3124846303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a80f5830 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fa798009fb0 tx=0x7fa798009b30).stop 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 -- 172.21.15.203:0/3124846303 shutdown_connections 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 --2- 172.21.15.203:0/3124846303 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a80f5830 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 -- 172.21.15.203:0/3124846303 >> 172.21.15.203:0/3124846303 conn(0x7fa7a80f0b70 msgr2=0x7fa7a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 -- 172.21.15.203:0/3124846303 shutdown_connections 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.830+0000 7fa7accc8700 1 -- 172.21.15.203:0/3124846303 wait complete. 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.831+0000 7fa7accc8700 1 Processor -- start 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.831+0000 7fa7accc8700 1 -- start start 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.831+0000 7fa7accc8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.831+0000 7fa7accc8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a8105bd0 con 0x7fa7a80f5410 2024-02-19T22:19:28.834 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7a659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:28.835 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7a659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:40424/0 (socket says 172.21.15.203:40424) 2024-02-19T22:19:28.835 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7a659c700 1 -- 172.21.15.203:0/2693360243 learned_addr learned my addr 172.21.15.203:0/2693360243 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:28.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7a659c700 1 -- 172.21.15.203:0/2693360243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798005040 con 0x7fa7a80f5410 2024-02-19T22:19:28.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7a659c700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa7980089f0 tx=0x7fa79800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:28.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa798009320 con 0x7fa7a80f5410 2024-02-19T22:19:28.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa798013380 con 0x7fa7a80f5410 2024-02-19T22:19:28.836 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a8105dd0 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa79801ed80 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.832+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a8106260 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.833+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa79801e420 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.833+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a8045ad0 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.833+0000 7fa796ffd700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 0x7fa790032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.833+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7980410d0 con 0x7fa7a80f5410 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.833+0000 7fa7a5d9b700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 0x7fa790032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:28.837 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.834+0000 7fa7a5d9b700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 0x7fa790032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa79c0096a0 tx=0x7fa79c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:28.840 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:28.838+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7980095f0 con 0x7fa7a80f5410 2024-02-19T22:19:29.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.043+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7a8107110 con 0x7fa7a80f5410 2024-02-19T22:19:29.046 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.044+0000 7fa796ffd700 1 -- 172.21.15.203:0/2693360243 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa79802c2d0 con 0x7fa7a80f5410 2024-02-19T22:19:29.046 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:29.046 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.045+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 msgr2=0x7fa790032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.045+0000 7fa7accc8700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 0x7fa790032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fa79c0096a0 tx=0x7fa79c006b40).stop 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.045+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 msgr2=0x7fa7a81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.045+0000 7fa7accc8700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fa7980089f0 tx=0x7fa79800b040).stop 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 shutdown_connections 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa79002ff60 0x7fa790032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 --2- 172.21.15.203:0/2693360243 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa7a80f5410 0x7fa7a81056e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 >> 172.21.15.203:0/2693360243 conn(0x7fa7a80f0b70 msgr2=0x7fa7a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:29.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 shutdown_connections 2024-02-19T22:19:29.049 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:29.046+0000 7fa7accc8700 1 -- 172.21.15.203:0/2693360243 wait complete. 2024-02-19T22:19:29.049 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:29.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:29 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:29.045027+0000 mon.smithi111 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.203:0/2693360243' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:29.214620+0000 mgr.smithi111.ftjzov (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:30.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:30.731 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:32.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 -- 172.21.15.203:0/776666070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 msgr2=0x7f65740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 --2- 172.21.15.203:0/776666070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f65740f55c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f6564009fb0 tx=0x7f6564009b30).stop 2024-02-19T22:19:32.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 -- 172.21.15.203:0/776666070 shutdown_connections 2024-02-19T22:19:32.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 --2- 172.21.15.203:0/776666070 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f65740f55c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 -- 172.21.15.203:0/776666070 >> 172.21.15.203:0/776666070 conn(0x7f65740f09a0 msgr2=0x7f65740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 -- 172.21.15.203:0/776666070 shutdown_connections 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.701+0000 7f6573fff700 1 -- 172.21.15.203:0/776666070 wait complete. 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6573fff700 1 Processor -- start 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6573fff700 1 -- start start 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6573fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.704 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6573fff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6574060280 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:40450/0 (socket says 172.21.15.203:40450) 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.702+0000 7f6572ffd700 1 -- 172.21.15.203:0/3182196507 learned_addr learned my addr 172.21.15.203:0/3182196507 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f6572ffd700 1 -- 172.21.15.203:0/3182196507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6564005040 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f6572ffd700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f65640089f0 tx=0x7f656400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6564009320 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6564013380 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574060480 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6564014e20 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.703+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65741099c0 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.704+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6564014420 con 0x7f65740f51a0 2024-02-19T22:19:32.706 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.704+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6574045ad0 con 0x7f65740f51a0 2024-02-19T22:19:32.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.704+0000 7f655b7fe700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 0x7f655c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:32.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.704+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6564040f50 con 0x7f65740f51a0 2024-02-19T22:19:32.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.704+0000 7f65727fc700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 0x7f655c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:32.707 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.705+0000 7f65727fc700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 0x7f655c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f65680096a0 tx=0x7f6568006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:32.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:31.214990+0000 mgr.smithi111.ftjzov (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:32.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.709+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f656406d2b0 con 0x7f65740f51a0 2024-02-19T22:19:32.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.916+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65740fad30 con 0x7f65740f51a0 2024-02-19T22:19:32.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.917+0000 7f655b7fe700 1 -- 172.21.15.203:0/3182196507 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f65640095f0 con 0x7f65740f51a0 2024-02-19T22:19:32.918 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:32.918 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 msgr2=0x7f655c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 0x7f655c0323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f65680096a0 tx=0x7f6568006b40).stop 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 msgr2=0x7f6574109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f65640089f0 tx=0x7f656400b040).stop 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 shutdown_connections 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f655c02ff10 0x7f655c0323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 --2- 172.21.15.203:0/3182196507 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f65740f51a0 0x7f6574109290 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 >> 172.21.15.203:0/3182196507 conn(0x7f65740f09a0 msgr2=0x7f65740f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 shutdown_connections 2024-02-19T22:19:32.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:32.918+0000 7f6573fff700 1 -- 172.21.15.203:0/3182196507 wait complete. 2024-02-19T22:19:32.921 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:33.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:33 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:32.917116+0000 mon.smithi111 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.203:0/3182196507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:34.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:34.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:33.215231+0000 mgr.smithi111.ftjzov (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:36.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 -- 172.21.15.203:0/439309857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 msgr2=0x7fe0ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 --2- 172.21.15.203:0/439309857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fe09c009fb0 tx=0x7fe09c009b30).stop 2024-02-19T22:19:36.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 -- 172.21.15.203:0/439309857 shutdown_connections 2024-02-19T22:19:36.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 --2- 172.21.15.203:0/439309857 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 -- 172.21.15.203:0/439309857 >> 172.21.15.203:0/439309857 conn(0x7fe0ac0f0b70 msgr2=0x7fe0ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 -- 172.21.15.203:0/439309857 shutdown_connections 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.551+0000 7fe0b3511700 1 -- 172.21.15.203:0/439309857 wait complete. 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b3511700 1 Processor -- start 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b3511700 1 -- start start 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b3511700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b3511700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac0fb960 con 0x7fe0ac0f5410 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b12ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b12ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53156/0 (socket says 172.21.15.203:53156) 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b12ad700 1 -- 172.21.15.203:0/2714231144 learned_addr learned my addr 172.21.15.203:0/2714231144 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b12ad700 1 -- 172.21.15.203:0/2714231144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c005040 con 0x7fe0ac0f5410 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.552+0000 7fe0b12ad700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe09c005010 tx=0x7fe09c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.553+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe09c009320 con 0x7fe0ac0f5410 2024-02-19T22:19:36.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.553+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac0fbb00 con 0x7fe0ac0f5410 2024-02-19T22:19:36.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.553+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe09c013380 con 0x7fe0ac0f5410 2024-02-19T22:19:36.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.553+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe09c014e50 con 0x7fe0ac0f5410 2024-02-19T22:19:36.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.553+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac0fbf90 con 0x7fe0ac0f5410 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe09c0095f0 con 0x7fe0ac0f5410 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac059100 con 0x7fe0ac0f5410 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0a1ffb700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 0x7fe0980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe09c040e70 con 0x7fe0ac0f5410 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0b0aac700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 0x7fe0980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:36.557 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.554+0000 7fe0b0aac700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 0x7fe0980323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe0a80096a0 tx=0x7fe0a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:36.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.558+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe09c014420 con 0x7fe0ac0f5410 2024-02-19T22:19:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:35.215579+0000 mgr.smithi111.ftjzov (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:36.762 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.761+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0ac045ad0 con 0x7fe0ac0f5410 2024-02-19T22:19:36.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.762+0000 7fe0a1ffb700 1 -- 172.21.15.203:0/2714231144 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe09c010230 con 0x7fe0ac0f5410 2024-02-19T22:19:36.763 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:36.763 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 msgr2=0x7fe0980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 0x7fe0980323d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fe0a80096a0 tx=0x7fe0a8006b40).stop 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 msgr2=0x7fe0ac0fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fe09c005010 tx=0x7fe09c00b040).stop 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 shutdown_connections 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe09802ff10 0x7fe0980323d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 --2- 172.21.15.203:0/2714231144 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe0ac0f5410 0x7fe0ac0fd4e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 >> 172.21.15.203:0/2714231144 conn(0x7fe0ac0f0b70 msgr2=0x7fe0ac05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 shutdown_connections 2024-02-19T22:19:36.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:36.763+0000 7fe0b3511700 1 -- 172.21.15.203:0/2714231144 wait complete. 2024-02-19T22:19:36.766 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:37.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:37 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:36.762121+0000 mon.smithi111 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.203:0/2714231144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:38.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:38.514 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:37.215948+0000 mgr.smithi111.ftjzov (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:40.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 -- 172.21.15.203:0/3343758529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 msgr2=0x7f999c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3343758529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c0f5830 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f998c009fb0 tx=0x7f998c009b30).stop 2024-02-19T22:19:40.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 -- 172.21.15.203:0/3343758529 shutdown_connections 2024-02-19T22:19:40.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3343758529 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c0f5830 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 -- 172.21.15.203:0/3343758529 >> 172.21.15.203:0/3343758529 conn(0x7f999c0f0b70 msgr2=0x7f999c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 -- 172.21.15.203:0/3343758529 shutdown_connections 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.411+0000 7f99a0d29700 1 -- 172.21.15.203:0/3343758529 wait complete. 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f99a0d29700 1 Processor -- start 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f99a0d29700 1 -- start start 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f99a0d29700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:40.415 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f99a0d29700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999c105c00 con 0x7f999c0f5410 2024-02-19T22:19:40.416 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f999a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f999a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53174/0 (socket says 172.21.15.203:53174) 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.412+0000 7f999a59c700 1 -- 172.21.15.203:0/3441541792 learned_addr learned my addr 172.21.15.203:0/3441541792 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f999a59c700 1 -- 172.21.15.203:0/3441541792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c005040 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f999a59c700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f998c0089f0 tx=0x7f998c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998c009320 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f999c105e00 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f998c013380 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998c01ed80 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.413+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999c106290 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.414+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999c045ad0 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.414+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f998c01e420 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.414+0000 7f998affd700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 0x7f9984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.414+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f998c0410d0 con 0x7f999c0f5410 2024-02-19T22:19:40.417 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.414+0000 7f9999d9b700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 0x7f9984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:40.418 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.415+0000 7f9999d9b700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 0x7f9984032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f99900096a0 tx=0x7f9990006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:40.421 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.419+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f998c01cb10 con 0x7f999c0f5410 2024-02-19T22:19:40.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.622+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f999c107140 con 0x7f999c0f5410 2024-02-19T22:19:40.624 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.622+0000 7f998affd700 1 -- 172.21.15.203:0/3441541792 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f998c004490 con 0x7f999c0f5410 2024-02-19T22:19:40.625 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:40.625 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 msgr2=0x7f9984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 0x7f9984032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f99900096a0 tx=0x7f9990006b40).stop 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 msgr2=0x7f999c105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f998c0089f0 tx=0x7f998c00b040).stop 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 shutdown_connections 2024-02-19T22:19:40.626 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f998402ff60 0x7f9984032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 --2- 172.21.15.203:0/3441541792 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f999c0f5410 0x7f999c105710 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:40.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 >> 172.21.15.203:0/3441541792 conn(0x7f999c0f0b70 msgr2=0x7f999c0f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:40.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 shutdown_connections 2024-02-19T22:19:40.627 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:40.624+0000 7f99a0d29700 1 -- 172.21.15.203:0/3441541792 wait complete. 2024-02-19T22:19:40.627 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:39.216255+0000 mgr.smithi111.ftjzov (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:41.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:41 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:40.623716+0000 mon.smithi111 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.203:0/3441541792' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:42.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:42.386 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:42.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:41.216493+0000 mgr.smithi111.ftjzov (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:44.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 -- 172.21.15.203:0/4019192764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 msgr2=0x7fd97c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 --2- 172.21.15.203:0/4019192764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c0f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fd970009fb0 tx=0x7fd970009b30).stop 2024-02-19T22:19:44.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 -- 172.21.15.203:0/4019192764 shutdown_connections 2024-02-19T22:19:44.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 --2- 172.21.15.203:0/4019192764 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c0f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 -- 172.21.15.203:0/4019192764 >> 172.21.15.203:0/4019192764 conn(0x7fd97c0f1c70 msgr2=0x7fd97c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.291+0000 7fd982f3a700 1 -- 172.21.15.203:0/4019192764 shutdown_connections 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.292+0000 7fd982f3a700 1 -- 172.21.15.203:0/4019192764 wait complete. 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.292+0000 7fd982f3a700 1 Processor -- start 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.292+0000 7fd982f3a700 1 -- start start 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.292+0000 7fd982f3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.292+0000 7fd982f3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c108f00 con 0x7fd97c0f6510 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd980cd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:44.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd980cd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:53196/0 (socket says 172.21.15.203:53196) 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd980cd6700 1 -- 172.21.15.203:0/299252231 learned_addr learned my addr 172.21.15.203:0/299252231 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd980cd6700 1 -- 172.21.15.203:0/299252231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd970005040 con 0x7fd97c0f6510 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd980cd6700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd9700089f0 tx=0x7fd97000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd970009320 con 0x7fd97c0f6510 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd970013380 con 0x7fd97c0f6510 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd97001ed80 con 0x7fd97c0f6510 2024-02-19T22:19:44.300 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c109100 con 0x7fd97c0f6510 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.293+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c109590 con 0x7fd97c0f6510 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.294+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd97001e420 con 0x7fd97c0f6510 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.294+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c045ad0 con 0x7fd97c0f6510 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.294+0000 7fd9797fa700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 0x7fd96c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.294+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd970040f00 con 0x7fd97c0f6510 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.295+0000 7fd97bfff700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 0x7fd96c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:44.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.295+0000 7fd97bfff700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 0x7fd96c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:44.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.299+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9700095f0 con 0x7fd97c0f6510 2024-02-19T22:19:44.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.505+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd97c10a440 con 0x7fd97c0f6510 2024-02-19T22:19:44.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.505+0000 7fd9797fa700 1 -- 172.21.15.203:0/299252231 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd97001cb10 con 0x7fd97c0f6510 2024-02-19T22:19:44.508 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:44.508 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 msgr2=0x7fd96c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 0x7fd96c032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).stop 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 msgr2=0x7fd97c108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fd9700089f0 tx=0x7fd97000b040).stop 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 shutdown_connections 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd96c02ff60 0x7fd96c032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 --2- 172.21.15.203:0/299252231 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd97c0f6510 0x7fd97c108a10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:44.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 >> 172.21.15.203:0/299252231 conn(0x7fd97c0f1c70 msgr2=0x7fd97c0f2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:44.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 shutdown_connections 2024-02-19T22:19:44.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:44.507+0000 7fd982f3a700 1 -- 172.21.15.203:0/299252231 wait complete. 2024-02-19T22:19:44.510 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:43.216767+0000 mgr.smithi111.ftjzov (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:45.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:44.506728+0000 mon.smithi111 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.203:0/299252231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:46.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:46.225 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:45.217098+0000 mgr.smithi111.ftjzov (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.196+0000 7f548b59e700 1 -- 172.21.15.203:0/2524151729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 msgr2=0x7f548c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.196+0000 7f548b59e700 1 --2- 172.21.15.203:0/2524151729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c0f66c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f5484009fb0 tx=0x7f5484009b30).stop 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.196+0000 7f548b59e700 1 -- 172.21.15.203:0/2524151729 shutdown_connections 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.196+0000 7f548b59e700 1 --2- 172.21.15.203:0/2524151729 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c0f66c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.196+0000 7f548b59e700 1 -- 172.21.15.203:0/2524151729 >> 172.21.15.203:0/2524151729 conn(0x7f548c0f1ac0 msgr2=0x7f548c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 -- 172.21.15.203:0/2524151729 shutdown_connections 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 -- 172.21.15.203:0/2524151729 wait complete. 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 Processor -- start 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 -- start start 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.197+0000 7f548b59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c106ab0 con 0x7f548c0f62a0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55768/0 (socket says 172.21.15.203:55768) 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548a59c700 1 -- 172.21.15.203:0/2514613039 learned_addr learned my addr 172.21.15.203:0/2514613039 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548a59c700 1 -- 172.21.15.203:0/2514613039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484005040 con 0x7f548c0f62a0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548a59c700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f54840089f0 tx=0x7f548400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5484009320 con 0x7f548c0f62a0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c106cb0 con 0x7f548c0f62a0 2024-02-19T22:19:48.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5484013380 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5484014e20 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.198+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c107140 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.199+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5484014420 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.199+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c045ad0 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.199+0000 7f5482ffd700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 0x7f54780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.199+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5484040f50 con 0x7f548c0f62a0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.199+0000 7f5489d9b700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 0x7f54780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:48.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.200+0000 7f5489d9b700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 0x7f54780323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f54740096a0 tx=0x7f5474006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:48.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.204+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f548406d2b0 con 0x7f548c0f62a0 2024-02-19T22:19:48.411 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.409+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f548c107ff0 con 0x7f548c0f62a0 2024-02-19T22:19:48.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.410+0000 7f5482ffd700 1 -- 172.21.15.203:0/2514613039 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f54840095f0 con 0x7f548c0f62a0 2024-02-19T22:19:48.412 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:48.412 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 msgr2=0x7f54780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 0x7f54780323d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f54740096a0 tx=0x7f5474006b40).stop 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 msgr2=0x7f548c1065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f54840089f0 tx=0x7f548400b040).stop 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 shutdown_connections 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f547802ff10 0x7f54780323d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 --2- 172.21.15.203:0/2514613039 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f548c0f62a0 0x7f548c1065c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 >> 172.21.15.203:0/2514613039 conn(0x7f548c0f1ac0 msgr2=0x7f548c0f24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 shutdown_connections 2024-02-19T22:19:48.414 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:48.411+0000 7f548b59e700 1 -- 172.21.15.203:0/2514613039 wait complete. 2024-02-19T22:19:48.415 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:47.217497+0000 mgr.smithi111.ftjzov (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:49.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:49 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:48.411076+0000 mon.smithi111 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.203:0/2514613039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:50.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:50.172 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:49.217742+0000 mgr.smithi111.ftjzov (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:52.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 -- 172.21.15.203:0/2808012009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 msgr2=0x7feff80f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 --2- 172.21.15.203:0/2808012009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff80f4c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fefec009fb0 tx=0x7fefec009b30).stop 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 -- 172.21.15.203:0/2808012009 shutdown_connections 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 --2- 172.21.15.203:0/2808012009 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff80f4c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 -- 172.21.15.203:0/2808012009 >> 172.21.15.203:0/2808012009 conn(0x7feff80f0260 msgr2=0x7feff80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 -- 172.21.15.203:0/2808012009 shutdown_connections 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.292+0000 7feffc812700 1 -- 172.21.15.203:0/2808012009 wait complete. 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.293+0000 7feffc812700 1 Processor -- start 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.293+0000 7feffc812700 1 -- start start 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.293+0000 7feffc812700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.296 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.293+0000 7feffc812700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8106e90 con 0x7feff80f47e0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feff659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feff659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55780/0 (socket says 172.21.15.203:55780) 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feff659c700 1 -- 172.21.15.203:0/594330925 learned_addr learned my addr 172.21.15.203:0/594330925 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feff659c700 1 -- 172.21.15.203:0/594330925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec005040 con 0x7feff80f47e0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feff659c700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fefec0089f0 tx=0x7fefec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefec009320 con 0x7feff80f47e0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefec013380 con 0x7feff80f47e0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefec014e20 con 0x7feff80f47e0 2024-02-19T22:19:52.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff8107090 con 0x7feff80f47e0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.294+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff8107520 con 0x7feff80f47e0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.295+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefec0095f0 con 0x7feff80f47e0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.295+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff8045ad0 con 0x7feff80f47e0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.295+0000 7fefe6ffd700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 0x7fefe00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.295+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefec040f10 con 0x7feff80f47e0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.295+0000 7feff5d9b700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 0x7fefe00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:52.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.296+0000 7feff5d9b700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 0x7fefe00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fefe80096a0 tx=0x7fefe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:52.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.300+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefec014420 con 0x7feff80f47e0 2024-02-19T22:19:52.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.507+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feff805eb50 con 0x7feff80f47e0 2024-02-19T22:19:52.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.508+0000 7fefe6ffd700 1 -- 172.21.15.203:0/594330925 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefec01d980 con 0x7feff80f47e0 2024-02-19T22:19:52.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:52.511 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:52.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 msgr2=0x7fefe00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 0x7fefe00323d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fefe80096a0 tx=0x7fefe8006b40).stop 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 msgr2=0x7feff81069a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fefec0089f0 tx=0x7fefec00b040).stop 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 shutdown_connections 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fefe002ff10 0x7fefe00323d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 --2- 172.21.15.203:0/594330925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7feff80f47e0 0x7feff81069a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 >> 172.21.15.203:0/594330925 conn(0x7feff80f0260 msgr2=0x7feff80f0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 shutdown_connections 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:52.510+0000 7feffc812700 1 -- 172.21.15.203:0/594330925 wait complete. 2024-02-19T22:19:52.513 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:52.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:51.218015+0000 mgr.smithi111.ftjzov (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:53.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:52.509300+0000 mon.smithi111 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.203:0/594330925' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:54.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:54.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:53.218270+0000 mgr.smithi111.ftjzov (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:56.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 -- 172.21.15.203:0/2854455229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 msgr2=0x7f25a80f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.373 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 --2- 172.21.15.203:0/2854455229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a80f4be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f259c009fb0 tx=0x7f259c009b30).stop 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 -- 172.21.15.203:0/2854455229 shutdown_connections 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 --2- 172.21.15.203:0/2854455229 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a80f4be0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 -- 172.21.15.203:0/2854455229 >> 172.21.15.203:0/2854455229 conn(0x7f25a80f02e0 msgr2=0x7f25a80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 -- 172.21.15.203:0/2854455229 shutdown_connections 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.370+0000 7f25aed86700 1 -- 172.21.15.203:0/2854455229 wait complete. 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.371+0000 7f25aed86700 1 Processor -- start 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.371+0000 7f25aed86700 1 -- start start 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.371+0000 7f25aed86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.374 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.371+0000 7f25aed86700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a8106cc0 con 0x7f25a80f47c0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25acb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25acb22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33788/0 (socket says 172.21.15.203:33788) 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25acb22700 1 -- 172.21.15.203:0/1526012842 learned_addr learned my addr 172.21.15.203:0/1526012842 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25acb22700 1 -- 172.21.15.203:0/1526012842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c005040 con 0x7f25a80f47c0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25acb22700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f259c005010 tx=0x7f259c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f259c009320 con 0x7f25a80f47c0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a8106ec0 con 0x7f25a80f47c0 2024-02-19T22:19:56.376 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f259c013380 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f259c014e50 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.372+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a8107350 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.373+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f259c0095f0 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.373+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a8045ad0 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.373+0000 7f25a57fa700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 0x7f25980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.373+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f259c040f60 con 0x7f25a80f47c0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.373+0000 7f25a7fff700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 0x7f25980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:19:56.377 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.374+0000 7f25a7fff700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 0x7f25980323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f25940096a0 tx=0x7f2594006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:19:56.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.378+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f259c014420 con 0x7f25a80f47c0 2024-02-19T22:19:56.588 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.586+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25a8108200 con 0x7f25a80f47c0 2024-02-19T22:19:56.589 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.586+0000 7f25a57fa700 1 -- 172.21.15.203:0/1526012842 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f259c010230 con 0x7f25a80f47c0 2024-02-19T22:19:56.589 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:19:56.589 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 msgr2=0x7f25980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 0x7f25980323d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f25940096a0 tx=0x7f2594006b40).stop 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 msgr2=0x7f25a81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f259c005010 tx=0x7f259c00b040).stop 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 shutdown_connections 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f259802ff10 0x7f25980323d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 --2- 172.21.15.203:0/1526012842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f25a80f47c0 0x7f25a81067d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 >> 172.21.15.203:0/1526012842 conn(0x7f25a80f02e0 msgr2=0x7f25a80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 shutdown_connections 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:19:56.588+0000 7f25aed86700 1 -- 172.21.15.203:0/1526012842 wait complete. 2024-02-19T22:19:56.591 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:19:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:55.218604+0000 mgr.smithi111.ftjzov (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:19:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:19:56.587687+0000 mon.smithi111 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.203:0/1526012842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:19:58.228 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:19:58.228 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:19:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:19:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:57.218880+0000 mgr.smithi111.ftjzov (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:00.232 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 -- 172.21.15.203:0/4196578868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 msgr2=0x7f44340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 --2- 172.21.15.203:0/4196578868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44340f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f4428009fb0 tx=0x7f4428009b30).stop 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 -- 172.21.15.203:0/4196578868 shutdown_connections 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 --2- 172.21.15.203:0/4196578868 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44340f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 -- 172.21.15.203:0/4196578868 >> 172.21.15.203:0/4196578868 conn(0x7f44340f0b70 msgr2=0x7f44340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 -- 172.21.15.203:0/4196578868 shutdown_connections 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 -- 172.21.15.203:0/4196578868 wait complete. 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.226+0000 7f4438ba0700 1 Processor -- start 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4438ba0700 1 -- start start 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4438ba0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4438ba0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4434105bd0 con 0x7f44340f5410 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f443259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.233 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f443259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33804/0 (socket says 172.21.15.203:33804) 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f443259c700 1 -- 172.21.15.203:0/1346827100 learned_addr learned my addr 172.21.15.203:0/1346827100 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f443259c700 1 -- 172.21.15.203:0/1346827100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4428005040 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f443259c700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4428005010 tx=0x7f442800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4428009320 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4434105dd0 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4428013380 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4428014e50 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.227+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4434106260 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.228+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4434045ad0 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.228+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4428014420 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.229+0000 7f4422ffd700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 0x7f441c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.229+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4428040e60 con 0x7f44340f5410 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.229+0000 7f4431d9b700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 0x7f441c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:00.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.229+0000 7f4431d9b700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 0x7f441c0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:00.236 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.233+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44280095f0 con 0x7f44340f5410 2024-02-19T22:20:00.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.442+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4434107110 con 0x7f44340f5410 2024-02-19T22:20:00.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.443+0000 7f4422ffd700 1 -- 172.21.15.203:0/1346827100 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f442801db70 con 0x7f44340f5410 2024-02-19T22:20:00.445 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:00.446 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 msgr2=0x7f441c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 0x7f441c0323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f44240096a0 tx=0x7f4424006b40).stop 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 msgr2=0x7f44341056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f4428005010 tx=0x7f442800b040).stop 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 shutdown_connections 2024-02-19T22:20:00.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f441c02ff10 0x7f441c0323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 --2- 172.21.15.203:0/1346827100 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f44340f5410 0x7f44341056e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:00.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 >> 172.21.15.203:0/1346827100 conn(0x7f44340f0b70 msgr2=0x7f44340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:00.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 shutdown_connections 2024-02-19T22:20:00.448 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:00.445+0000 7f4438ba0700 1 -- 172.21.15.203:0/1346827100 wait complete. 2024-02-19T22:20:00.448 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:19:59.219211+0000 mgr.smithi111.ftjzov (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:01.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:00.444400+0000 mon.smithi111 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.203:0/1346827100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:02.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:02.221 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:01.219443+0000 mgr.smithi111.ftjzov (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.075+0000 7f72821dc700 1 -- 172.21.15.203:0/4164422373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 msgr2=0x7f727c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.075+0000 7f72821dc700 1 --2- 172.21.15.203:0/4164422373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c0f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f7270009fb0 tx=0x7f7270009b30).stop 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 -- 172.21.15.203:0/4164422373 shutdown_connections 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 --2- 172.21.15.203:0/4164422373 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c0f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 -- 172.21.15.203:0/4164422373 >> 172.21.15.203:0/4164422373 conn(0x7f727c0f1c70 msgr2=0x7f727c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.078 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 -- 172.21.15.203:0/4164422373 shutdown_connections 2024-02-19T22:20:04.079 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 -- 172.21.15.203:0/4164422373 wait complete. 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.076+0000 7f72821dc700 1 Processor -- start 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f72821dc700 1 -- start start 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f72821dc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f72821dc700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c106d00 con 0x7f727c0f6510 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f727b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f727b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33820/0 (socket says 172.21.15.203:33820) 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f727b7fe700 1 -- 172.21.15.203:0/2069083921 learned_addr learned my addr 172.21.15.203:0/2069083921 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f727b7fe700 1 -- 172.21.15.203:0/2069083921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270005040 con 0x7f727c0f6510 2024-02-19T22:20:04.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.077+0000 7f727b7fe700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7270005010 tx=0x7f727000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7270009320 con 0x7f727c0f6510 2024-02-19T22:20:04.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c106f00 con 0x7f727c0f6510 2024-02-19T22:20:04.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7270013380 con 0x7f727c0f6510 2024-02-19T22:20:04.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7270014e50 con 0x7f727c0f6510 2024-02-19T22:20:04.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c107390 con 0x7f727c0f6510 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72700095f0 con 0x7f727c0f6510 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.078+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c045ad0 con 0x7f727c0f6510 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.079+0000 7f7263fff700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 0x7f72640323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.079+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7270040f10 con 0x7f727c0f6510 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.079+0000 7f727affd700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 0x7f72640323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:04.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.079+0000 7f727affd700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 0x7f72640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f726c0096a0 tx=0x7f726c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:04.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.083+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7270014420 con 0x7f727c0f6510 2024-02-19T22:20:04.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.289+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f727c108240 con 0x7f727c0f6510 2024-02-19T22:20:04.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.289+0000 7f7263fff700 1 -- 172.21.15.203:0/2069083921 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7270010230 con 0x7f727c0f6510 2024-02-19T22:20:04.292 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:04.292 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 msgr2=0x7f72640323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 0x7f72640323d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f726c0096a0 tx=0x7f726c006b40).stop 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 msgr2=0x7f727c106810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f7270005010 tx=0x7f727000b040).stop 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 shutdown_connections 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f726402ff10 0x7f72640323d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 --2- 172.21.15.203:0/2069083921 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f727c0f6510 0x7f727c106810 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 >> 172.21.15.203:0/2069083921 conn(0x7f727c0f1c70 msgr2=0x7f727c0f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 shutdown_connections 2024-02-19T22:20:04.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:04.291+0000 7f72821dc700 1 -- 172.21.15.203:0/2069083921 wait complete. 2024-02-19T22:20:04.295 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:03.219769+0000 mgr.smithi111.ftjzov (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:05.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:04.290758+0000 mon.smithi111 (mon.0) 235 : audit [DBG] from='client.? 172.21.15.203:0/2069083921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:06.099 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:06.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:05.220165+0000 mgr.smithi111.ftjzov (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:07.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 -- 172.21.15.203:0/1060665005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 msgr2=0x7f63340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:07.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 --2- 172.21.15.203:0/1060665005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f63340f90f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f6324009fb0 tx=0x7f6324009b30).stop 2024-02-19T22:20:07.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 -- 172.21.15.203:0/1060665005 shutdown_connections 2024-02-19T22:20:07.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 --2- 172.21.15.203:0/1060665005 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f63340f90f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 -- 172.21.15.203:0/1060665005 >> 172.21.15.203:0/1060665005 conn(0x7f63340f0b70 msgr2=0x7f63340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 -- 172.21.15.203:0/1060665005 shutdown_connections 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.992+0000 7f633a91d700 1 -- 172.21.15.203:0/1060665005 wait complete. 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f633a91d700 1 Processor -- start 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f633a91d700 1 -- start start 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f633a91d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f633a91d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6334106c70 con 0x7f6334060280 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f6333fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f6333fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55924/0 (socket says 172.21.15.203:55924) 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.993+0000 7f6333fff700 1 -- 172.21.15.203:0/2717741528 learned_addr learned my addr 172.21.15.203:0/2717741528 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f6333fff700 1 -- 172.21.15.203:0/2717741528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324005040 con 0x7f6334060280 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f6333fff700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f63240089f0 tx=0x7f632400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6324009320 con 0x7f6334060280 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6334106e70 con 0x7f6334060280 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6324013380 con 0x7f6334060280 2024-02-19T22:20:07.997 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6324014e20 con 0x7f6334060280 2024-02-19T22:20:07.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.994+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6334107300 con 0x7f6334060280 2024-02-19T22:20:07.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63240263a0 con 0x7f6334060280 2024-02-19T22:20:07.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f6330ff9700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 0x7f631c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:07.998 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6324040ed0 con 0x7f6334060280 2024-02-19T22:20:07.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6334045ad0 con 0x7f6334060280 2024-02-19T22:20:07.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f63337fe700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 0x7f631c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:07.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:07.995+0000 7f63337fe700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 0x7f631c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f63280096a0 tx=0x7f6328006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:08.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.000+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6324012920 con 0x7f6334060280 2024-02-19T22:20:08.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.204+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6334108040 con 0x7f6334060280 2024-02-19T22:20:08.207 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.205+0000 7f6330ff9700 1 -- 172.21.15.203:0/2717741528 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f632402b020 con 0x7f6334060280 2024-02-19T22:20:08.207 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:08.207 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 msgr2=0x7f631c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 0x7f631c0323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f63280096a0 tx=0x7f6328006b40).stop 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 msgr2=0x7f6334106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f63240089f0 tx=0x7f632400b040).stop 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 shutdown_connections 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f631c02ff10 0x7f631c0323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 --2- 172.21.15.203:0/2717741528 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6334060280 0x7f6334106780 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 >> 172.21.15.203:0/2717741528 conn(0x7f63340f0b70 msgr2=0x7f63340f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.206+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 shutdown_connections 2024-02-19T22:20:08.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:08.207+0000 7f633a91d700 1 -- 172.21.15.203:0/2717741528 wait complete. 2024-02-19T22:20:08.210 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:08.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:07.220485+0000 mgr.smithi111.ftjzov (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:08.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:08.206163+0000 mon.smithi111 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.203:0/2717741528' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:09.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:09.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:10.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:09.220723+0000 mgr.smithi111.ftjzov (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 -- 172.21.15.203:0/1167855693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 msgr2=0x7f02a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 --2- 172.21.15.203:0/1167855693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a80f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f0298009fb0 tx=0x7f0298009b30).stop 2024-02-19T22:20:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 -- 172.21.15.203:0/1167855693 shutdown_connections 2024-02-19T22:20:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 --2- 172.21.15.203:0/1167855693 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a80f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:11.805 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 -- 172.21.15.203:0/1167855693 >> 172.21.15.203:0/1167855693 conn(0x7f02a80f1c70 msgr2=0x7f02a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 -- 172.21.15.203:0/1167855693 shutdown_connections 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.802+0000 7f02aff53700 1 -- 172.21.15.203:0/1167855693 wait complete. 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02aff53700 1 Processor -- start 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02aff53700 1 -- start start 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02aff53700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02aff53700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a8106cd0 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02adcef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02adcef700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55944/0 (socket says 172.21.15.203:55944) 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02adcef700 1 -- 172.21.15.203:0/1975154207 learned_addr learned my addr 172.21.15.203:0/1975154207 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02adcef700 1 -- 172.21.15.203:0/1975154207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298005040 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.803+0000 7f02adcef700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0298005010 tx=0x7f029800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0298009320 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8106ed0 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0298013380 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0298014e50 con 0x7f02a80f6510 2024-02-19T22:20:11.806 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a8107360 con 0x7f02a80f6510 2024-02-19T22:20:11.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.804+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02980095f0 con 0x7f02a80f6510 2024-02-19T22:20:11.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.805+0000 7f029e7fc700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 0x7f02940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:11.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.805+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a8045ad0 con 0x7f02a80f6510 2024-02-19T22:20:11.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.805+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0298040e70 con 0x7f02a80f6510 2024-02-19T22:20:11.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.805+0000 7f02ad4ee700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 0x7f02940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:11.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.805+0000 7f02ad4ee700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 0x7f02940323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f02a40096a0 tx=0x7f02a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:11.810 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:11.809+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f029801d990 con 0x7f02a80f6510 2024-02-19T22:20:12.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.013+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02a8108190 con 0x7f02a80f6510 2024-02-19T22:20:12.016 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.014+0000 7f029e7fc700 1 -- 172.21.15.203:0/1975154207 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0298010230 con 0x7f02a80f6510 2024-02-19T22:20:12.016 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:12.016 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:12.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 msgr2=0x7f02940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 0x7f02940323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f02a40096a0 tx=0x7f02a4006b40).stop 2024-02-19T22:20:12.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 msgr2=0x7f02a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:12.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f0298005010 tx=0x7f029800b040).stop 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 shutdown_connections 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f029402ff10 0x7f02940323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 --2- 172.21.15.203:0/1975154207 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02a80f6510 0x7f02a81067e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 >> 172.21.15.203:0/1975154207 conn(0x7f02a80f1c70 msgr2=0x7f02a80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 shutdown_connections 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:12.016+0000 7f02aff53700 1 -- 172.21.15.203:0/1975154207 wait complete. 2024-02-19T22:20:12.018 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:11.220946+0000 mgr.smithi111.ftjzov (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:12.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:12.014512+0000 mon.smithi111 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.203:0/1975154207' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:13.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:13.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:13.221214+0000 mgr.smithi111.ftjzov (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 -- 172.21.15.203:0/1811157450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 msgr2=0x7ff2040f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 --2- 172.21.15.203:0/1811157450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff2040f6db0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7ff1ec009fb0 tx=0x7ff1ec009b30).stop 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 -- 172.21.15.203:0/1811157450 shutdown_connections 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 --2- 172.21.15.203:0/1811157450 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff2040f6db0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 -- 172.21.15.203:0/1811157450 >> 172.21.15.203:0/1811157450 conn(0x7ff2040f0260 msgr2=0x7ff2040f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 -- 172.21.15.203:0/1811157450 shutdown_connections 2024-02-19T22:20:15.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.662+0000 7ff209109700 1 -- 172.21.15.203:0/1811157450 wait complete. 2024-02-19T22:20:15.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff209109700 1 Processor -- start 2024-02-19T22:20:15.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff209109700 1 -- start start 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff209109700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff209109700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040f52b0 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff202d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff202d9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:47894/0 (socket says 172.21.15.203:47894) 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.663+0000 7ff202d9d700 1 -- 172.21.15.203:0/2762456002 learned_addr learned my addr 172.21.15.203:0/2762456002 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff202d9d700 1 -- 172.21.15.203:0/2762456002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec005040 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff202d9d700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff1ec005010 tx=0x7ff1ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1ec009320 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040f54b0 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1ec013380 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1ec014e50 con 0x7ff2040f6990 2024-02-19T22:20:15.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.664+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff204109bc0 con 0x7ff2040f6990 2024-02-19T22:20:15.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.665+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1ec0095f0 con 0x7ff2040f6990 2024-02-19T22:20:15.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.665+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff204059100 con 0x7ff2040f6990 2024-02-19T22:20:15.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.665+0000 7ff1fb7fe700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 0x7ff1f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:15.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.665+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1ec040e70 con 0x7ff2040f6990 2024-02-19T22:20:15.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.665+0000 7ff20259c700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 0x7ff1f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:15.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.666+0000 7ff20259c700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 0x7ff1f00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff1f40096a0 tx=0x7ff1f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:15.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.670+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1ec014420 con 0x7ff2040f6990 2024-02-19T22:20:15.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.878+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff204045ad0 con 0x7ff2040f6990 2024-02-19T22:20:15.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.878+0000 7ff1fb7fe700 1 -- 172.21.15.203:0/2762456002 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff1ec010230 con 0x7ff2040f6990 2024-02-19T22:20:15.880 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:15.880 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:15.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 msgr2=0x7ff1f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 0x7ff1f00323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7ff1f40096a0 tx=0x7ff1f4006b40).stop 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 msgr2=0x7ff204109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7ff1ec005010 tx=0x7ff1ec00b040).stop 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 shutdown_connections 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff1f002ff10 0x7ff1f00323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 --2- 172.21.15.203:0/2762456002 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff2040f6990 0x7ff204109510 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 >> 172.21.15.203:0/2762456002 conn(0x7ff2040f0260 msgr2=0x7ff2040f0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 shutdown_connections 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:15.880+0000 7ff209109700 1 -- 172.21.15.203:0/2762456002 wait complete. 2024-02-19T22:20:15.882 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:15.221521+0000 mgr.smithi111.ftjzov (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:15.878690+0000 mon.smithi111 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.203:0/2762456002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:17.628 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:17.629 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:17.221791+0000 mgr.smithi111.ftjzov (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:17.302287+0000 mon.smithi111 (mon.0) 239 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:20:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:17.304082+0000 mon.smithi111 (mon.0) 240 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- 172.21.15.203:0/4165097604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 msgr2=0x7f13600f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 --2- 172.21.15.203:0/4165097604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f13600f66c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f1350009fb0 tx=0x7f1350009b30).stop 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- 172.21.15.203:0/4165097604 shutdown_connections 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 --2- 172.21.15.203:0/4165097604 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f13600f66c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- 172.21.15.203:0/4165097604 >> 172.21.15.203:0/4165097604 conn(0x7f13600f1aa0 msgr2=0x7f13600f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- 172.21.15.203:0/4165097604 shutdown_connections 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- 172.21.15.203:0/4165097604 wait complete. 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 Processor -- start 2024-02-19T22:20:19.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- start start 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.566+0000 7f136528d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360069e20 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:47912/0 (socket says 172.21.15.203:47912) 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135ffff700 1 -- 172.21.15.203:0/1096395600 learned_addr learned my addr 172.21.15.203:0/1096395600 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135ffff700 1 -- 172.21.15.203:0/1096395600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1350005040 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135ffff700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1350009830 tx=0x7f135000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1350009320 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360069fc0 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1350013380 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1350014420 con 0x7f13600f62a0 2024-02-19T22:20:19.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.567+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136006a450 con 0x7f13600f62a0 2024-02-19T22:20:19.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.568+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360059100 con 0x7f13600f62a0 2024-02-19T22:20:19.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.568+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f135001e420 con 0x7f13600f62a0 2024-02-19T22:20:19.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.568+0000 7f135cff9700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:19.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.568+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1350040e30 con 0x7f13600f62a0 2024-02-19T22:20:19.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.569+0000 7f135f7fe700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:19.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.569+0000 7f135f7fe700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 0x7f1348032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f13540096a0 tx=0x7f1354006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:19.575 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.573+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f135001cb10 con 0x7f13600f62a0 2024-02-19T22:20:19.779 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.777+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1360045ad0 con 0x7f13600f62a0 2024-02-19T22:20:19.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.778+0000 7f135cff9700 1 -- 172.21.15.203:0/1096395600 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1350004490 con 0x7f13600f62a0 2024-02-19T22:20:19.781 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:19.781 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:19.781 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 msgr2=0x7f1348032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 0x7f1348032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f13540096a0 tx=0x7f1354006b40).stop 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 msgr2=0x7f136006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f1350009830 tx=0x7f135000b040).stop 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 shutdown_connections 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f134802ff60 0x7f1348032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 --2- 172.21.15.203:0/1096395600 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f13600f62a0 0x7f136006b9a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 >> 172.21.15.203:0/1096395600 conn(0x7f13600f1aa0 msgr2=0x7f13600f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 shutdown_connections 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:19.779+0000 7f136528d700 1 -- 172.21.15.203:0/1096395600 wait complete. 2024-02-19T22:20:19.782 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:19.222132+0000 mgr.smithi111.ftjzov (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:19.778954+0000 mon.smithi111 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.203:0/1096395600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:21.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:21.495 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:22.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:21.222402+0000 mgr.smithi111.ftjzov (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.502+0000 7fed64186700 1 -- 172.21.15.203:0/819781777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 msgr2=0x7fed5c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.502+0000 7fed64186700 1 --2- 172.21.15.203:0/819781777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c0f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fed50009fb0 tx=0x7fed50009b30).stop 2024-02-19T22:20:23.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.502+0000 7fed64186700 1 -- 172.21.15.203:0/819781777 shutdown_connections 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.502+0000 7fed64186700 1 --2- 172.21.15.203:0/819781777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c0f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.502+0000 7fed64186700 1 -- 172.21.15.203:0/819781777 >> 172.21.15.203:0/819781777 conn(0x7fed5c0f1c70 msgr2=0x7fed5c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 -- 172.21.15.203:0/819781777 shutdown_connections 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 -- 172.21.15.203:0/819781777 wait complete. 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 Processor -- start 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 -- start start 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed64186700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c106cd0 con 0x7fed5c0f6510 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed61f22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed61f22700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:47936/0 (socket says 172.21.15.203:47936) 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.503+0000 7fed61f22700 1 -- 172.21.15.203:0/442531846 learned_addr learned my addr 172.21.15.203:0/442531846 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed61f22700 1 -- 172.21.15.203:0/442531846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed50005040 con 0x7fed5c0f6510 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed61f22700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fed50005010 tx=0x7fed5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:23.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed50009320 con 0x7fed5c0f6510 2024-02-19T22:20:23.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed5c106ed0 con 0x7fed5c0f6510 2024-02-19T22:20:23.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed50013380 con 0x7fed5c0f6510 2024-02-19T22:20:23.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed50014e50 con 0x7fed5c0f6510 2024-02-19T22:20:23.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.504+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed5c107360 con 0x7fed5c0f6510 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.505+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5c045ad0 con 0x7fed5c0f6510 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.505+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed500095f0 con 0x7fed5c0f6510 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.505+0000 7fed4e7fc700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 0x7fed480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.505+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed50040e90 con 0x7fed5c0f6510 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.505+0000 7fed61721700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 0x7fed480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:23.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.506+0000 7fed61721700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 0x7fed480323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fed580096a0 tx=0x7fed58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:23.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.510+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed50014420 con 0x7fed5c0f6510 2024-02-19T22:20:23.719 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.717+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed5c108210 con 0x7fed5c0f6510 2024-02-19T22:20:23.720 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.718+0000 7fed4e7fc700 1 -- 172.21.15.203:0/442531846 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed50010230 con 0x7fed5c0f6510 2024-02-19T22:20:23.720 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:23.720 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 msgr2=0x7fed480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 0x7fed480323d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fed580096a0 tx=0x7fed58006b40).stop 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 msgr2=0x7fed5c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fed50005010 tx=0x7fed5000b040).stop 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 shutdown_connections 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fed4802ff10 0x7fed480323d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 --2- 172.21.15.203:0/442531846 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fed5c0f6510 0x7fed5c1067e0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:23.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 >> 172.21.15.203:0/442531846 conn(0x7fed5c0f1c70 msgr2=0x7fed5c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:23.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 shutdown_connections 2024-02-19T22:20:23.723 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:23.719+0000 7fed64186700 1 -- 172.21.15.203:0/442531846 wait complete. 2024-02-19T22:20:23.723 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:23.222701+0000 mgr.smithi111.ftjzov (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:24.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:23.718986+0000 mon.smithi111 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.203:0/442531846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:25.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:25.406 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:25.223056+0000 mgr.smithi111.ftjzov (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 -- 172.21.15.203:0/3652274548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 msgr2=0x7f74580f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 --2- 172.21.15.203:0/3652274548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74580f55c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f744c009fb0 tx=0x7f744c009b30).stop 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 -- 172.21.15.203:0/3652274548 shutdown_connections 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 --2- 172.21.15.203:0/3652274548 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74580f55c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 -- 172.21.15.203:0/3652274548 >> 172.21.15.203:0/3652274548 conn(0x7f74580f09a0 msgr2=0x7f74580f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 -- 172.21.15.203:0/3652274548 shutdown_connections 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 -- 172.21.15.203:0/3652274548 wait complete. 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.549+0000 7f745cc90700 1 Processor -- start 2024-02-19T22:20:27.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f745cc90700 1 -- start start 2024-02-19T22:20:27.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f745cc90700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f745cc90700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74581059c0 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f74577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f74577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38962/0 (socket says 172.21.15.203:38962) 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f74577fe700 1 -- 172.21.15.203:0/2785333745 learned_addr learned my addr 172.21.15.203:0/2785333745 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f74577fe700 1 -- 172.21.15.203:0/2785333745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c005040 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.550+0000 7f74577fe700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f744c0089f0 tx=0x7f744c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c009320 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7458105bc0 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f744c013380 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f744c014e20 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7458106050 con 0x7f74580f51a0 2024-02-19T22:20:27.555 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.551+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f744c0095f0 con 0x7f74580f51a0 2024-02-19T22:20:27.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.552+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7458045ad0 con 0x7f74580f51a0 2024-02-19T22:20:27.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.552+0000 7f743ffff700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 0x7f74400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:27.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.552+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f744c040f10 con 0x7f74580f51a0 2024-02-19T22:20:27.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.552+0000 7f7456ffd700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 0x7f74400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:27.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.552+0000 7f7456ffd700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 0x7f74400323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f74480096a0 tx=0x7f7448006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:27.558 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.556+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f744c014420 con 0x7f74580f51a0 2024-02-19T22:20:27.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.765+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7458106f00 con 0x7f74580f51a0 2024-02-19T22:20:27.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.766+0000 7f743ffff700 1 -- 172.21.15.203:0/2785333745 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f744c01d980 con 0x7f74580f51a0 2024-02-19T22:20:27.768 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:27.769 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 msgr2=0x7f74400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 0x7f74400323d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f74480096a0 tx=0x7f7448006b40).stop 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 msgr2=0x7f74581054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f744c0089f0 tx=0x7f744c00b040).stop 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 shutdown_connections 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f744002ff10 0x7f74400323d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 --2- 172.21.15.203:0/2785333745 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f74580f51a0 0x7f74581054d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 >> 172.21.15.203:0/2785333745 conn(0x7f74580f09a0 msgr2=0x7f74580f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 shutdown_connections 2024-02-19T22:20:27.770 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:27.767+0000 7f745cc90700 1 -- 172.21.15.203:0/2785333745 wait complete. 2024-02-19T22:20:27.771 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:27.223315+0000 mgr.smithi111.ftjzov (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:28.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:27.766937+0000 mon.smithi111 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.203:0/2785333745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:29.425 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:29.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:29.223622+0000 mgr.smithi111.ftjzov (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:31.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 -- 172.21.15.203:0/3425892167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 msgr2=0x7f80cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 --2- 172.21.15.203:0/3425892167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc0f55c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f80bc009fb0 tx=0x7f80bc009b30).stop 2024-02-19T22:20:31.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 -- 172.21.15.203:0/3425892167 shutdown_connections 2024-02-19T22:20:31.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 --2- 172.21.15.203:0/3425892167 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc0f55c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 -- 172.21.15.203:0/3425892167 >> 172.21.15.203:0/3425892167 conn(0x7f80cc0f09a0 msgr2=0x7f80cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:31.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 -- 172.21.15.203:0/3425892167 shutdown_connections 2024-02-19T22:20:31.299 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.295+0000 7f80cbfff700 1 -- 172.21.15.203:0/3425892167 wait complete. 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.296+0000 7f80cbfff700 1 Processor -- start 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.296+0000 7f80cbfff700 1 -- start start 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80cbfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80cc060280 con 0x7f80cc0f51a0 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80caffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80caffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38976/0 (socket says 172.21.15.203:38976) 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80caffd700 1 -- 172.21.15.203:0/1254585110 learned_addr learned my addr 172.21.15.203:0/1254585110 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80caffd700 1 -- 172.21.15.203:0/1254585110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80bc005040 con 0x7f80cc0f51a0 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80caffd700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f80bc0089f0 tx=0x7f80bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80bc009320 con 0x7f80cc0f51a0 2024-02-19T22:20:31.301 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80bc013380 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80cc060480 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80bc014e20 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.297+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80cc1099c0 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.298+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80bc0095f0 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.298+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc045ad0 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.298+0000 7f80b37fe700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 0x7f80b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.298+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80bc040f50 con 0x7f80cc0f51a0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.299+0000 7f80ca7fc700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 0x7f80b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:31.302 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.299+0000 7f80ca7fc700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 0x7f80b40323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f80c00096a0 tx=0x7f80c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:31.305 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.303+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80bc014420 con 0x7f80cc0f51a0 2024-02-19T22:20:31.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.507+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80cc0fad30 con 0x7f80cc0f51a0 2024-02-19T22:20:31.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.508+0000 7f80b37fe700 1 -- 172.21.15.203:0/1254585110 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80bc01d980 con 0x7f80cc0f51a0 2024-02-19T22:20:31.511 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:31.511 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 msgr2=0x7f80b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 0x7f80b40323d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f80c00096a0 tx=0x7f80c0006b40).stop 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 msgr2=0x7f80cc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f80bc0089f0 tx=0x7f80bc00b040).stop 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 shutdown_connections 2024-02-19T22:20:31.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f80b402ff10 0x7f80b40323d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 --2- 172.21.15.203:0/1254585110 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f80cc0f51a0 0x7f80cc109290 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:31.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 >> 172.21.15.203:0/1254585110 conn(0x7f80cc0f09a0 msgr2=0x7f80cc0f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:31.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 shutdown_connections 2024-02-19T22:20:31.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:31.510+0000 7f80cbfff700 1 -- 172.21.15.203:0/1254585110 wait complete. 2024-02-19T22:20:31.513 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:32.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:31.223875+0000 mgr.smithi111.ftjzov (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:31.509382+0000 mon.smithi111 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.203:0/1254585110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:33.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:33.174 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:34.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:33.224215+0000 mgr.smithi111.ftjzov (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 -- 172.21.15.203:0/3791198925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 msgr2=0x7f151c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3791198925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f150c009fb0 tx=0x7f150c009b30).stop 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 -- 172.21.15.203:0/3791198925 shutdown_connections 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3791198925 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 -- 172.21.15.203:0/3791198925 >> 172.21.15.203:0/3791198925 conn(0x7f151c0f1c70 msgr2=0x7f151c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:35.250 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 -- 172.21.15.203:0/3791198925 shutdown_connections 2024-02-19T22:20:35.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.247+0000 7f1520fbe700 1 -- 172.21.15.203:0/3791198925 wait complete. 2024-02-19T22:20:35.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f1520fbe700 1 Processor -- start 2024-02-19T22:20:35.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f1520fbe700 1 -- start start 2024-02-19T22:20:35.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f1520fbe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:35.251 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f1520fbe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c106cd0 con 0x7f151c0f6510 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f151a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f151a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39000/0 (socket says 172.21.15.203:39000) 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.248+0000 7f151a59c700 1 -- 172.21.15.203:0/3199960742 learned_addr learned my addr 172.21.15.203:0/3199960742 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f151a59c700 1 -- 172.21.15.203:0/3199960742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150c005040 con 0x7f151c0f6510 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f151a59c700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f150c005010 tx=0x7f150c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:35.252 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f150c009320 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c106ed0 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f150c013380 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f150c014e50 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.249+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c107360 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.250+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f150c0095f0 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.250+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c045ad0 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.250+0000 7f150affd700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 0x7f15040323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.250+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f150c040e90 con 0x7f151c0f6510 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.250+0000 7f1519d9b700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 0x7f15040323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:35.253 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.251+0000 7f1519d9b700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 0x7f15040323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f15100096a0 tx=0x7f1510006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:35.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.255+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f150c014420 con 0x7f151c0f6510 2024-02-19T22:20:35.465 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.463+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f151c108210 con 0x7f151c0f6510 2024-02-19T22:20:35.466 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.464+0000 7f150affd700 1 -- 172.21.15.203:0/3199960742 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f150c010230 con 0x7f151c0f6510 2024-02-19T22:20:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:35.467 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:35.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.465+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 msgr2=0x7f15040323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:35.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.465+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 0x7f15040323d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f15100096a0 tx=0x7f1510006b40).stop 2024-02-19T22:20:35.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 msgr2=0x7f151c1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f150c005010 tx=0x7f150c00b040).stop 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 shutdown_connections 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f150402ff10 0x7f15040323d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 --2- 172.21.15.203:0/3199960742 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f151c0f6510 0x7f151c1067e0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 >> 172.21.15.203:0/3199960742 conn(0x7f151c0f1c70 msgr2=0x7f151c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 shutdown_connections 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:35.466+0000 7f1520fbe700 1 -- 172.21.15.203:0/3199960742 wait complete. 2024-02-19T22:20:35.469 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:35.224562+0000 mgr.smithi111.ftjzov (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:35.465238+0000 mon.smithi111 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.203:0/3199960742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:37.120 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:37.121 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:38.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:37.224943+0000 mgr.smithi111.ftjzov (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:39.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 -- 172.21.15.203:0/2646137960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c0f6510 msgr2=0x7f708c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:39.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 --2- 172.21.15.203:0/2646137960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c0f6510 0x7f708c0f6930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f7084009fb0 tx=0x7f7084009b30).stop 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 -- 172.21.15.203:0/2646137960 shutdown_connections 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 --2- 172.21.15.203:0/2646137960 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c0f6510 0x7f708c0f6930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 -- 172.21.15.203:0/2646137960 >> 172.21.15.203:0/2646137960 conn(0x7f708c0f1c70 msgr2=0x7f708c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 -- 172.21.15.203:0/2646137960 shutdown_connections 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.116+0000 7f7092180700 1 -- 172.21.15.203:0/2646137960 wait complete. 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f7092180700 1 Processor -- start 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f7092180700 1 -- start start 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f7092180700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:39.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f7092180700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f708c1071f0 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f708b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.117+0000 7f708b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46308/0 (socket says 172.21.15.203:46308) 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f708b7fe700 1 -- 172.21.15.203:0/2849919765 learned_addr learned my addr 172.21.15.203:0/2849919765 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f708b7fe700 1 -- 172.21.15.203:0/2849919765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7084005040 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f708b7fe700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f7084009f80 tx=0x7f7084009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7084013380 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f708c1073f0 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7084013790 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7084014420 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.118+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f708c107f40 con 0x7f708c1068e0 2024-02-19T22:20:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.119+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7084025060 con 0x7f708c1068e0 2024-02-19T22:20:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.119+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f708c059100 con 0x7f708c1068e0 2024-02-19T22:20:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.119+0000 7f7083fff700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 0x7f7078032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.119+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7084040e00 con 0x7f708c1068e0 2024-02-19T22:20:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.119+0000 7f708affd700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 0x7f7078032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.120+0000 7f708affd700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 0x7f7078032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f70740096a0 tx=0x7f7074006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:39.126 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.124+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f708400f280 con 0x7f708c1068e0 2024-02-19T22:20:39.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.332+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f708c108690 con 0x7f708c1068e0 2024-02-19T22:20:39.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.333+0000 7f7083fff700 1 -- 172.21.15.203:0/2849919765 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f708402d2d0 con 0x7f708c1068e0 2024-02-19T22:20:39.335 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:39.335 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.334+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 msgr2=0x7f7078032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 0x7f7078032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f70740096a0 tx=0x7f7074006b40).stop 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 msgr2=0x7f708c106d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f7084009f80 tx=0x7f7084009b30).stop 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 shutdown_connections 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f707802ff60 0x7f7078032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 --2- 172.21.15.203:0/2849919765 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f708c1068e0 0x7f708c106d00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:39.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 >> 172.21.15.203:0/2849919765 conn(0x7f708c0f1c70 msgr2=0x7f708c0f27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:39.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 shutdown_connections 2024-02-19T22:20:39.338 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:39.335+0000 7f7092180700 1 -- 172.21.15.203:0/2849919765 wait complete. 2024-02-19T22:20:39.338 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:40.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:39.225302+0000 mgr.smithi111.ftjzov (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:40.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:39.334155+0000 mon.smithi111 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.203:0/2849919765' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:41.003 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:41.004 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:42.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:41.225592+0000 mgr.smithi111.ftjzov (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- 172.21.15.203:0/1472551565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 msgr2=0x7fa6380f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 --2- 172.21.15.203:0/1472551565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6380f66c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fa62c009fb0 tx=0x7fa62c009b30).stop 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- 172.21.15.203:0/1472551565 shutdown_connections 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 --2- 172.21.15.203:0/1472551565 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6380f66c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- 172.21.15.203:0/1472551565 >> 172.21.15.203:0/1472551565 conn(0x7fa6380f1aa0 msgr2=0x7fa6380f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- 172.21.15.203:0/1472551565 shutdown_connections 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- 172.21.15.203:0/1472551565 wait complete. 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 Processor -- start 2024-02-19T22:20:42.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.881+0000 7fa63d999700 1 -- start start 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63d999700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63d999700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa638106ac0 con 0x7fa6380f62a0 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63c997700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63c997700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46324/0 (socket says 172.21.15.203:46324) 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63c997700 1 -- 172.21.15.203:0/305266741 learned_addr learned my addr 172.21.15.203:0/305266741 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63c997700 1 -- 172.21.15.203:0/305266741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c005040 con 0x7fa6380f62a0 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.882+0000 7fa63c997700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa62c0089f0 tx=0x7fa62c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:42.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa62c009320 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa638106cc0 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa62c013380 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa62c014e20 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa638107150 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa62c0095f0 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.883+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa638045ad0 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.884+0000 7fa6357fa700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 0x7fa6280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.884+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa62c040f10 con 0x7fa6380f62a0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.884+0000 7fa637fff700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 0x7fa6280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:42.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.884+0000 7fa637fff700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 0x7fa6280323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa6240096a0 tx=0x7fa624006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:42.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:42.888+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa62c014420 con 0x7fa6380f62a0 2024-02-19T22:20:43.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.093+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa638108000 con 0x7fa6380f62a0 2024-02-19T22:20:43.096 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.093+0000 7fa6357fa700 1 -- 172.21.15.203:0/305266741 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa62c01d980 con 0x7fa6380f62a0 2024-02-19T22:20:43.096 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:43.096 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:43.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 msgr2=0x7fa6280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:43.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 0x7fa6280323d0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa6240096a0 tx=0x7fa624006b40).stop 2024-02-19T22:20:43.097 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 msgr2=0x7fa6381065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa62c0089f0 tx=0x7fa62c00b040).stop 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 shutdown_connections 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa62802ff10 0x7fa6280323d0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 --2- 172.21.15.203:0/305266741 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa6380f62a0 0x7fa6381065d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 >> 172.21.15.203:0/305266741 conn(0x7fa6380f1aa0 msgr2=0x7fa6380f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 shutdown_connections 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:43.095+0000 7fa63d999700 1 -- 172.21.15.203:0/305266741 wait complete. 2024-02-19T22:20:43.098 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:43.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:43 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:43.094717+0000 mon.smithi111 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.203:0/305266741' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:43.225882+0000 mgr.smithi111.ftjzov (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:44.844 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:44.845 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:45.226213+0000 mgr.smithi111.ftjzov (mgr.14162) 230 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:46.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 -- 172.21.15.203:0/969665413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 msgr2=0x7efe380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:46.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 --2- 172.21.15.203:0/969665413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe380f5830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7efe2c009fb0 tx=0x7efe2c009b30).stop 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 -- 172.21.15.203:0/969665413 shutdown_connections 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 --2- 172.21.15.203:0/969665413 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe380f5830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 -- 172.21.15.203:0/969665413 >> 172.21.15.203:0/969665413 conn(0x7efe380f0b70 msgr2=0x7efe380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 -- 172.21.15.203:0/969665413 shutdown_connections 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.779+0000 7efe3da95700 1 -- 172.21.15.203:0/969665413 wait complete. 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe3da95700 1 Processor -- start 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe3da95700 1 -- start start 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe3da95700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe3da95700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe38105bd0 con 0x7efe380f5410 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:46.784 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe377fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51768/0 (socket says 172.21.15.203:51768) 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.780+0000 7efe377fe700 1 -- 172.21.15.203:0/3316636019 learned_addr learned my addr 172.21.15.203:0/3316636019 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe377fe700 1 -- 172.21.15.203:0/3316636019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c005040 con 0x7efe380f5410 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe377fe700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efe2c005010 tx=0x7efe2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe2c009320 con 0x7efe380f5410 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe38105dd0 con 0x7efe380f5410 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efe2c013380 con 0x7efe380f5410 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe2c014e50 con 0x7efe380f5410 2024-02-19T22:20:46.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.781+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe38106260 con 0x7efe380f5410 2024-02-19T22:20:46.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.782+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe2c0095f0 con 0x7efe380f5410 2024-02-19T22:20:46.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.782+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38045ad0 con 0x7efe380f5410 2024-02-19T22:20:46.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.782+0000 7efe1ffff700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 0x7efe200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:46.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.782+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe2c040e90 con 0x7efe380f5410 2024-02-19T22:20:46.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.782+0000 7efe36ffd700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 0x7efe200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:46.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.783+0000 7efe36ffd700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 0x7efe200323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efe280096a0 tx=0x7efe28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:46.789 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.787+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe2c014420 con 0x7efe380f5410 2024-02-19T22:20:46.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.990+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe38107110 con 0x7efe380f5410 2024-02-19T22:20:46.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.991+0000 7efe1ffff700 1 -- 172.21.15.203:0/3316636019 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe2c010230 con 0x7efe380f5410 2024-02-19T22:20:46.994 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:46.994 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 msgr2=0x7efe200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 0x7efe200323d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efe280096a0 tx=0x7efe28006b40).stop 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 msgr2=0x7efe381056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7efe2c005010 tx=0x7efe2c00b040).stop 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 shutdown_connections 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7efe2002ff10 0x7efe200323d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 --2- 172.21.15.203:0/3316636019 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7efe380f5410 0x7efe381056e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 >> 172.21.15.203:0/3316636019 conn(0x7efe380f0b70 msgr2=0x7efe380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 shutdown_connections 2024-02-19T22:20:46.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:46.993+0000 7efe3da95700 1 -- 172.21.15.203:0/3316636019 wait complete. 2024-02-19T22:20:46.997 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:47.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:47 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:46.992582+0000 mon.smithi111 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.203:0/3316636019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:47.226586+0000 mgr.smithi111.ftjzov (mgr.14162) 231 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:48.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:48.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:50.679 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 -- 172.21.15.203:0/4185299838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 msgr2=0x7fe9280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/4185299838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9280f66c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fe918009fb0 tx=0x7fe918009b30).stop 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 -- 172.21.15.203:0/4185299838 shutdown_connections 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/4185299838 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9280f66c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 -- 172.21.15.203:0/4185299838 >> 172.21.15.203:0/4185299838 conn(0x7fe9280f1aa0 msgr2=0x7fe9280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 -- 172.21.15.203:0/4185299838 shutdown_connections 2024-02-19T22:20:50.680 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.678+0000 7fe92ebbe700 1 -- 172.21.15.203:0/4185299838 wait complete. 2024-02-19T22:20:50.681 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92ebbe700 1 Processor -- start 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92ebbe700 1 -- start start 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92ebbe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92ebbe700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe928106ac0 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92dbbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92dbbc700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51794/0 (socket says 172.21.15.203:51794) 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92dbbc700 1 -- 172.21.15.203:0/1138097170 learned_addr learned my addr 172.21.15.203:0/1138097170 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.679+0000 7fe92dbbc700 1 -- 172.21.15.203:0/1138097170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe918005040 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe92dbbc700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fe918009830 tx=0x7fe91800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe918009320 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe928106cc0 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe918013380 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe918014420 con 0x7fe9280f62a0 2024-02-19T22:20:50.682 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.680+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe928107150 con 0x7fe9280f62a0 2024-02-19T22:20:50.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe91801e420 con 0x7fe9280f62a0 2024-02-19T22:20:50.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe928045ad0 con 0x7fe9280f62a0 2024-02-19T22:20:50.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe91e7fc700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 0x7fe914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:50.683 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe918040e30 con 0x7fe9280f62a0 2024-02-19T22:20:50.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe92d3bb700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 0x7fe914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:50.684 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.681+0000 7fe92d3bb700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 0x7fe914032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:50.687 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.686+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe91801cb10 con 0x7fe9280f62a0 2024-02-19T22:20:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:49.226892+0000 mgr.smithi111.ftjzov (mgr.14162) 232 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:50.891 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.890+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9280fbe30 con 0x7fe9280f62a0 2024-02-19T22:20:50.892 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.891+0000 7fe91e7fc700 1 -- 172.21.15.203:0/1138097170 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe918004490 con 0x7fe9280f62a0 2024-02-19T22:20:50.892 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:50.892 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 msgr2=0x7fe914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 0x7fe914032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe9240096a0 tx=0x7fe924006b40).stop 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 msgr2=0x7fe9281065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fe918009830 tx=0x7fe91800b040).stop 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 shutdown_connections 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe91402ff60 0x7fe914032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 --2- 172.21.15.203:0/1138097170 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe9280f62a0 0x7fe9281065d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 >> 172.21.15.203:0/1138097170 conn(0x7fe9280f1aa0 msgr2=0x7fe9280f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 shutdown_connections 2024-02-19T22:20:50.894 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:50.892+0000 7fe92ebbe700 1 -- 172.21.15.203:0/1138097170 wait complete. 2024-02-19T22:20:50.895 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:51.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:51 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:50.891181+0000 mon.smithi111 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.203:0/1138097170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:52.661 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:52.662 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:52.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:51.227250+0000 mgr.smithi111.ftjzov (mgr.14162) 233 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:54.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 -- 172.21.15.203:0/3151709717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 msgr2=0x7f4f3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 --2- 172.21.15.203:0/3151709717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c0f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f4f2c009fb0 tx=0x7f4f2c009b30).stop 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 -- 172.21.15.203:0/3151709717 shutdown_connections 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 --2- 172.21.15.203:0/3151709717 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c0f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 -- 172.21.15.203:0/3151709717 >> 172.21.15.203:0/3151709717 conn(0x7f4f3c0f1c70 msgr2=0x7f4f3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 -- 172.21.15.203:0/3151709717 shutdown_connections 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.689+0000 7f4f43382700 1 -- 172.21.15.203:0/3151709717 wait complete. 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f43382700 1 Processor -- start 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f43382700 1 -- start start 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f43382700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f43382700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3c069e20 con 0x7f4f3c0f6510 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f4111e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:54.693 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f4111e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:51818/0 (socket says 172.21.15.203:51818) 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f4111e700 1 -- 172.21.15.203:0/3961940175 learned_addr learned my addr 172.21.15.203:0/3961940175 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.690+0000 7f4f4111e700 1 -- 172.21.15.203:0/3961940175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c005040 con 0x7f4f3c0f6510 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f4111e700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f4f2c0089f0 tx=0x7f4f2c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f2c009320 con 0x7f4f3c0f6510 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3c069fc0 con 0x7f4f3c0f6510 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4f2c013380 con 0x7f4f3c0f6510 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f2c01ed80 con 0x7f4f3c0f6510 2024-02-19T22:20:54.694 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.691+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3c06a450 con 0x7f4f3c0f6510 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.692+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4f2c01e420 con 0x7f4f3c0f6510 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.692+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f3c059100 con 0x7f4f3c0f6510 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.692+0000 7f4f31ffb700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 0x7f4f28032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.692+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4f2c040fc0 con 0x7f4f3c0f6510 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.692+0000 7f4f4091d700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 0x7f4f28032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:54.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.693+0000 7f4f4091d700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 0x7f4f28032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:54.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.697+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4f2c06d2b0 con 0x7f4f3c0f6510 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:53.227535+0000 mgr.smithi111.ftjzov (mgr.14162) 234 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:20:53.498085+0000 mgr.smithi111.ftjzov (mgr.14162) 235 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi111 ... 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Verifying port 9100 ... 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 1 from systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: systemctl: stderr Job for ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111.service failed because a timeout was exceeded. 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: systemctl: stderr See "systemctl status ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111.service" and "journalctl -xe" for details. 2024-02-19T22:20:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ports=daemon_ports) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:20:54.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: Failed command: systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: yield (conn, connr) 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: code, '\n'.join(err))) 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi111 ... 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Verifying port 9100 ... 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 1 from systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: systemctl: stderr Job for ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111.service failed because a timeout was exceeded. 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: systemctl: stderr See "systemctl status ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111.service" and "journalctl -xe" for details. 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:20:54.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: ports=daemon_ports) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: c, osd_fsid=osd_fsid, ports=ports) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: Failed command: systemctl start ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@node-exporter.smithi111 2024-02-19T22:20:54.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:20:53.503595+0000 mgr.smithi111.ftjzov (mgr.14162) 236 : cephadm [INF] Deploying daemon prometheus.smithi111 on smithi111 2024-02-19T22:20:54.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.904+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4f3c0fc0a0 con 0x7f4f3c0f6510 2024-02-19T22:20:54.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.905+0000 7f4f31ffb700 1 -- 172.21.15.203:0/3961940175 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4f2c0095f0 con 0x7f4f3c0f6510 2024-02-19T22:20:54.906 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:54.907 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.906+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 msgr2=0x7f4f28032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.906+0000 7f4f43382700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 0x7f4f28032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f4f380096a0 tx=0x7f4f38006b40).stop 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.906+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 msgr2=0x7f4f3c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f4f2c0089f0 tx=0x7f4f2c00b040).stop 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 shutdown_connections 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4f2802ff60 0x7f4f28032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:54.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 --2- 172.21.15.203:0/3961940175 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4f3c0f6510 0x7f4f3c06b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:54.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 >> 172.21.15.203:0/3961940175 conn(0x7f4f3c0f1c70 msgr2=0x7f4f3c0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:54.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 shutdown_connections 2024-02-19T22:20:54.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:54.907+0000 7f4f43382700 1 -- 172.21.15.203:0/3961940175 wait complete. 2024-02-19T22:20:54.909 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:55.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:54.905159+0000 mon.smithi111 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.203:0/3961940175' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:20:56.566 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:20:56.566 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:20:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:55.227958+0000 mgr.smithi111.ftjzov (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 -- 172.21.15.203:0/1298642187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 msgr2=0x7f4df00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/1298642187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df00f4b60 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f4de4009fb0 tx=0x7f4de4009b30).stop 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 -- 172.21.15.203:0/1298642187 shutdown_connections 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/1298642187 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df00f4b60 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 -- 172.21.15.203:0/1298642187 >> 172.21.15.203:0/1298642187 conn(0x7f4df00f0260 msgr2=0x7f4df00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 -- 172.21.15.203:0/1298642187 shutdown_connections 2024-02-19T22:20:58.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.519+0000 7f4df4ae2700 1 -- 172.21.15.203:0/1298642187 wait complete. 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4df4ae2700 1 Processor -- start 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4df4ae2700 1 -- start start 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4df4ae2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4df4ae2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df0106ca0 con 0x7f4df00f4740 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4dee59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4dee59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:57752/0 (socket says 172.21.15.203:57752) 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4dee59c700 1 -- 172.21.15.203:0/3319415049 learned_addr learned my addr 172.21.15.203:0/3319415049 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4dee59c700 1 -- 172.21.15.203:0/3319415049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4005040 con 0x7f4df00f4740 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4dee59c700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4de40089f0 tx=0x7f4de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.520+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de4009320 con 0x7f4df00f4740 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4de4013380 con 0x7f4df00f4740 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df0106ea0 con 0x7f4df00f4740 2024-02-19T22:20:58.524 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de401ed80 con 0x7f4df00f4740 2024-02-19T22:20:58.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df0107330 con 0x7f4df00f4740 2024-02-19T22:20:58.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4de401e420 con 0x7f4df00f4740 2024-02-19T22:20:58.525 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.521+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0045ad0 con 0x7f4df00f4740 2024-02-19T22:20:58.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.522+0000 7f4ddeffd700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:20:58.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.522+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4de4041010 con 0x7f4df00f4740 2024-02-19T22:20:58.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.522+0000 7f4dedd9b700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:20:58.526 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.522+0000 7f4dedd9b700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 0x7f4dd8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4de00096a0 tx=0x7f4de0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:20:58.528 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.526+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4de40095f0 con 0x7f4df00f4740 2024-02-19T22:20:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:57.228320+0000 mgr.smithi111.ftjzov (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:20:58.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.734+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4df01081e0 con 0x7f4df00f4740 2024-02-19T22:20:58.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.735+0000 7f4ddeffd700 1 -- 172.21.15.203:0/3319415049 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4de402c2d0 con 0x7f4df00f4740 2024-02-19T22:20:58.737 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:20:58.737 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 msgr2=0x7f4dd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 0x7f4dd8032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f4de00096a0 tx=0x7f4de0006b40).stop 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 msgr2=0x7f4df01067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f4de40089f0 tx=0x7f4de400b040).stop 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 shutdown_connections 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4dd802ff60 0x7f4dd8032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 --2- 172.21.15.203:0/3319415049 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4df00f4740 0x7f4df01067b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 >> 172.21.15.203:0/3319415049 conn(0x7f4df00f0260 msgr2=0x7f4df00f0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:20:58.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 shutdown_connections 2024-02-19T22:20:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:20:58.736+0000 7f4df4ae2700 1 -- 172.21.15.203:0/3319415049 wait complete. 2024-02-19T22:20:58.740 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:20:59.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:20:59 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:20:58.736062+0000 mon.smithi111 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.203:0/3319415049' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:00.512 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:00.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:20:59.228652+0000 mgr.smithi111.ftjzov (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:02.352 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 -- 172.21.15.203:0/4120850655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 msgr2=0x7f29440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 --2- 172.21.15.203:0/4120850655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f29440f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f2934009fb0 tx=0x7f2934009b30).stop 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 -- 172.21.15.203:0/4120850655 shutdown_connections 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 --2- 172.21.15.203:0/4120850655 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f29440f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 -- 172.21.15.203:0/4120850655 >> 172.21.15.203:0/4120850655 conn(0x7f29440f0b70 msgr2=0x7f29440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 -- 172.21.15.203:0/4120850655 shutdown_connections 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.348+0000 7f294c134700 1 -- 172.21.15.203:0/4120850655 wait complete. 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f294c134700 1 Processor -- start 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f294c134700 1 -- start start 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f294c134700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f294c134700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2944060240 con 0x7f29440f5410 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f2949ed0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f2949ed0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:57770/0 (socket says 172.21.15.203:57770) 2024-02-19T22:21:02.353 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.349+0000 7f2949ed0700 1 -- 172.21.15.203:0/1563802407 learned_addr learned my addr 172.21.15.203:0/1563802407 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f2949ed0700 1 -- 172.21.15.203:0/1563802407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934005040 con 0x7f29440f5410 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f2949ed0700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2934005010 tx=0x7f293400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2934009320 con 0x7f29440f5410 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2944060440 con 0x7f29440f5410 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2934013380 con 0x7f29440f5410 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2934014e50 con 0x7f29440f5410 2024-02-19T22:21:02.354 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.350+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2944109bc0 con 0x7f29440f5410 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.351+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2934014420 con 0x7f29440f5410 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.351+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2944045ad0 con 0x7f29440f5410 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.351+0000 7f293a7fc700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 0x7f29300323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.351+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2934040e30 con 0x7f29440f5410 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.351+0000 7f29496cf700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 0x7f29300323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:02.355 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.352+0000 7f29496cf700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 0x7f29300323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f29400096a0 tx=0x7f2940006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:02.358 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.356+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29340095f0 con 0x7f29440f5410 2024-02-19T22:21:02.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.564+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f294410a3e0 con 0x7f29440f5410 2024-02-19T22:21:02.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.565+0000 7f293a7fc700 1 -- 172.21.15.203:0/1563802407 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f293401db70 con 0x7f29440f5410 2024-02-19T22:21:02.567 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:02.568 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:02.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 msgr2=0x7f29300323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 0x7f29300323d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f29400096a0 tx=0x7f2940006b40).stop 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 msgr2=0x7f2944109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f2934005010 tx=0x7f293400b040).stop 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 shutdown_connections 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f293002ff10 0x7f29300323d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 --2- 172.21.15.203:0/1563802407 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f29440f5410 0x7f2944109510 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 >> 172.21.15.203:0/1563802407 conn(0x7f29440f0b70 msgr2=0x7f29440f15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 shutdown_connections 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:02.567+0000 7f294c134700 1 -- 172.21.15.203:0/1563802407 wait complete. 2024-02-19T22:21:02.570 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:01.228920+0000 mgr.smithi111.ftjzov (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:03.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:02.566343+0000 mon.smithi111 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.203:0/1563802407' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:04.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:04.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:04.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:03.229161+0000 mgr.smithi111.ftjzov (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:06.173 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 -- 172.21.15.203:0/1090723071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 msgr2=0x7f590c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 --2- 172.21.15.203:0/1090723071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c0f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f5900009fb0 tx=0x7f5900009b30).stop 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 -- 172.21.15.203:0/1090723071 shutdown_connections 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 --2- 172.21.15.203:0/1090723071 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c0f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 -- 172.21.15.203:0/1090723071 >> 172.21.15.203:0/1090723071 conn(0x7f590c0f0b70 msgr2=0x7f590c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 -- 172.21.15.203:0/1090723071 shutdown_connections 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.170+0000 7f591126a700 1 -- 172.21.15.203:0/1090723071 wait complete. 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f591126a700 1 Processor -- start 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f591126a700 1 -- start start 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f591126a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f591126a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590c060220 con 0x7f590c0f5410 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f590affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f590affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:36272/0 (socket says 172.21.15.203:36272) 2024-02-19T22:21:06.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.171+0000 7f590affd700 1 -- 172.21.15.203:0/542997332 learned_addr learned my addr 172.21.15.203:0/542997332 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f590affd700 1 -- 172.21.15.203:0/542997332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900005040 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f590affd700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f5900005010 tx=0x7f590000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5900009320 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590c060420 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5900013380 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5900014e50 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.172+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590c109bb0 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5900014420 con 0x7f590c0f5410 2024-02-19T22:21:06.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590c059100 con 0x7f590c0f5410 2024-02-19T22:21:06.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f58f37fe700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 0x7f58f40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:06.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5900040e30 con 0x7f590c0f5410 2024-02-19T22:21:06.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f590a7fc700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 0x7f58f40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:06.177 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.173+0000 7f590a7fc700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 0x7f58f40323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f58fc0096a0 tx=0x7f58fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:06.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.177+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59000095f0 con 0x7f590c0f5410 2024-02-19T22:21:06.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.383+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f590c045ad0 con 0x7f590c0f5410 2024-02-19T22:21:06.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.384+0000 7f58f37fe700 1 -- 172.21.15.203:0/542997332 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f590001db70 con 0x7f590c0f5410 2024-02-19T22:21:06.386 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:06.386 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 msgr2=0x7f58f40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 0x7f58f40323d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f58fc0096a0 tx=0x7f58fc006b40).stop 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 msgr2=0x7f590c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f5900005010 tx=0x7f590000b040).stop 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 shutdown_connections 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f58f402ff10 0x7f58f40323d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 --2- 172.21.15.203:0/542997332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f590c0f5410 0x7f590c109500 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 >> 172.21.15.203:0/542997332 conn(0x7f590c0f0b70 msgr2=0x7f590c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 shutdown_connections 2024-02-19T22:21:06.388 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:06.385+0000 7f591126a700 1 -- 172.21.15.203:0/542997332 wait complete. 2024-02-19T22:21:06.389 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:05.229553+0000 mgr.smithi111.ftjzov (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:07.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:07 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:06.385060+0000 mon.smithi111 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.203:0/542997332' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:08.223 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:08.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:07.229870+0000 mgr.smithi111.ftjzov (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:10.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.248+0000 7f0493d68700 1 -- 172.21.15.203:0/2282302716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 msgr2=0x7f048c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:10.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.248+0000 7f0493d68700 1 --2- 172.21.15.203:0/2282302716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c0f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f047c009fb0 tx=0x7f047c009b30).stop 2024-02-19T22:21:10.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.248+0000 7f0493d68700 1 -- 172.21.15.203:0/2282302716 shutdown_connections 2024-02-19T22:21:10.284 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.248+0000 7f0493d68700 1 --2- 172.21.15.203:0/2282302716 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c0f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.248+0000 7f0493d68700 1 -- 172.21.15.203:0/2282302716 >> 172.21.15.203:0/2282302716 conn(0x7f048c0f0b70 msgr2=0x7f048c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 -- 172.21.15.203:0/2282302716 shutdown_connections 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 -- 172.21.15.203:0/2282302716 wait complete. 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 Processor -- start 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 -- start start 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0493d68700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048c060220 con 0x7f048c0f5410 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0491b04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0491b04700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:36292/0 (socket says 172.21.15.203:36292) 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.249+0000 7f0491b04700 1 -- 172.21.15.203:0/2040328653 learned_addr learned my addr 172.21.15.203:0/2040328653 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f0491b04700 1 -- 172.21.15.203:0/2040328653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f047c005040 con 0x7f048c0f5410 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f0491b04700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f047c005010 tx=0x7f047c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c009320 con 0x7f048c0f5410 2024-02-19T22:21:10.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048c060420 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048c109bf0 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f047c013380 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.250+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f047c014df0 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.251+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048c059100 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.251+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f047c014420 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.251+0000 7f04827fc700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 0x7f04780323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.251+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f047c0410e0 con 0x7f048c0f5410 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.251+0000 7f0491303700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 0x7f04780323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.252+0000 7f0491303700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 0x7f04780323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04880096a0 tx=0x7f0488006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:10.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.256+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f047c00fb30 con 0x7f048c0f5410 2024-02-19T22:21:10.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.466+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f048c045ad0 con 0x7f048c0f5410 2024-02-19T22:21:10.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.467+0000 7f04827fc700 1 -- 172.21.15.203:0/2040328653 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f047c00fb30 con 0x7f048c0f5410 2024-02-19T22:21:10.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:10.469 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.468+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 msgr2=0x7f04780323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.468+0000 7f0493d68700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 0x7f04780323d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f04880096a0 tx=0x7f0488006b40).stop 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.468+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 msgr2=0x7f048c109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.468+0000 7f0493d68700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f047c005010 tx=0x7f047c00b040).stop 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 shutdown_connections 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f047802ff10 0x7f04780323d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:10.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 --2- 172.21.15.203:0/2040328653 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f048c0f5410 0x7f048c109540 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:10.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 >> 172.21.15.203:0/2040328653 conn(0x7f048c0f0b70 msgr2=0x7f048c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:10.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 shutdown_connections 2024-02-19T22:21:10.472 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:10.469+0000 7f0493d68700 1 -- 172.21.15.203:0/2040328653 wait complete. 2024-02-19T22:21:10.472 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:09.230229+0000 mgr.smithi111.ftjzov (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:11.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:10.468133+0000 mon.smithi111 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.203:0/2040328653' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:12.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:12.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:11.230485+0000 mgr.smithi111.ftjzov (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:14.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- 172.21.15.203:0/687714928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 msgr2=0x7ff8080f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:14.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 --2- 172.21.15.203:0/687714928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8080f55c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7ff804009fb0 tx=0x7ff804009b30).stop 2024-02-19T22:21:14.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- 172.21.15.203:0/687714928 shutdown_connections 2024-02-19T22:21:14.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 --2- 172.21.15.203:0/687714928 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8080f55c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:14.040 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- 172.21.15.203:0/687714928 >> 172.21.15.203:0/687714928 conn(0x7ff8080f09a0 msgr2=0x7ff8080f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- 172.21.15.203:0/687714928 shutdown_connections 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- 172.21.15.203:0/687714928 wait complete. 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 Processor -- start 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- start start 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.037+0000 7ff80ec88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff808106ac0 con 0x7ff8080f51a0 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80dc86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80dc86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:36306/0 (socket says 172.21.15.203:36306) 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80dc86700 1 -- 172.21.15.203:0/1122626560 learned_addr learned my addr 172.21.15.203:0/1122626560 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80dc86700 1 -- 172.21.15.203:0/1122626560 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804005040 con 0x7ff8080f51a0 2024-02-19T22:21:14.041 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80dc86700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff8040089f0 tx=0x7ff80400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:14.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff804009320 con 0x7ff8080f51a0 2024-02-19T22:21:14.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff808106cc0 con 0x7ff8080f51a0 2024-02-19T22:21:14.042 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff808107150 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff804013380 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.038+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff804014db0 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.039+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f0004fa0 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.039+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff804014420 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.039+0000 7ff7fe7fc700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 0x7ff7f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.039+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff804041100 con 0x7ff8080f51a0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.040+0000 7ff80d485700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 0x7ff7f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:14.043 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.040+0000 7ff80d485700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 0x7ff7f4032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:14.046 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.044+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff804012d00 con 0x7ff8080f51a0 2024-02-19T22:21:14.254 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.252+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7f0005670 con 0x7ff8080f51a0 2024-02-19T22:21:14.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.253+0000 7ff7fe7fc700 1 -- 172.21.15.203:0/1122626560 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff804012b20 con 0x7ff8080f51a0 2024-02-19T22:21:14.255 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:14.255 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 msgr2=0x7ff7f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 0x7ff7f4032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ff7f80096a0 tx=0x7ff7f8006b40).stop 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 msgr2=0x7ff8081065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7ff8040089f0 tx=0x7ff80400b040).stop 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 shutdown_connections 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff7f402ff60 0x7ff7f4032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 --2- 172.21.15.203:0/1122626560 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff8080f51a0 0x7ff8081065d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:14.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.254+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 >> 172.21.15.203:0/1122626560 conn(0x7ff8080f09a0 msgr2=0x7ff80805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:14.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.255+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 shutdown_connections 2024-02-19T22:21:14.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:14.255+0000 7ff80ec88700 1 -- 172.21.15.203:0/1122626560 wait complete. 2024-02-19T22:21:14.258 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:13.230764+0000 mgr.smithi111.ftjzov (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:15.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:15 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:14.254218+0000 mon.smithi111 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.203:0/1122626560' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:15.998 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:15.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:15.231030+0000 mgr.smithi111.ftjzov (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 -- 172.21.15.203:0/4194802348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 msgr2=0x7f15f00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 --2- 172.21.15.203:0/4194802348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f00f66c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f15e4009fb0 tx=0x7f15e4009b30).stop 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 -- 172.21.15.203:0/4194802348 shutdown_connections 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 --2- 172.21.15.203:0/4194802348 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f00f66c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 -- 172.21.15.203:0/4194802348 >> 172.21.15.203:0/4194802348 conn(0x7f15f00f1aa0 msgr2=0x7f15f00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 -- 172.21.15.203:0/4194802348 shutdown_connections 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 -- 172.21.15.203:0/4194802348 wait complete. 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.847+0000 7f15f516d700 1 Processor -- start 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15f516d700 1 -- start start 2024-02-19T22:21:17.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15f516d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15f516d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f0069e20 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15effff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41398/0 (socket says 172.21.15.203:41398) 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.848+0000 7f15effff700 1 -- 172.21.15.203:0/1417724299 learned_addr learned my addr 172.21.15.203:0/1417724299 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15effff700 1 -- 172.21.15.203:0/1417724299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e4005040 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15effff700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f15e4009830 tx=0x7f15e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e4009320 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15e4013380 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f0069fc0 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15e4014420 con 0x7f15f00f62a0 2024-02-19T22:21:17.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.849+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f006a450 con 0x7f15f00f62a0 2024-02-19T22:21:17.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15e401e420 con 0x7f15f00f62a0 2024-02-19T22:21:17.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15ecff9700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:17.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f0059100 con 0x7f15f00f62a0 2024-02-19T22:21:17.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15e4040eb0 con 0x7f15f00f62a0 2024-02-19T22:21:17.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15ef7fe700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:17.855 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.850+0000 7f15ef7fe700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 0x7f15d8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:17.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:17.854+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15e40020d0 con 0x7f15f00f62a0 2024-02-19T22:21:18.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.057+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15f0045ad0 con 0x7f15f00f62a0 2024-02-19T22:21:18.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.058+0000 7f15ecff9700 1 -- 172.21.15.203:0/1417724299 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f15e4004490 con 0x7f15f00f62a0 2024-02-19T22:21:18.061 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:18.061 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:18.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 msgr2=0x7f15d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:18.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 0x7f15d8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f15e00096a0 tx=0x7f15e0006b40).stop 2024-02-19T22:21:18.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 msgr2=0x7f15f006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:18.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f15e4009830 tx=0x7f15e400b040).stop 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 shutdown_connections 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f15d802ff60 0x7f15d8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 --2- 172.21.15.203:0/1417724299 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f15f00f62a0 0x7f15f006b9a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.059+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 >> 172.21.15.203:0/1417724299 conn(0x7f15f00f1aa0 msgr2=0x7f15f00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.060+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 shutdown_connections 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:18.060+0000 7f15f516d700 1 -- 172.21.15.203:0/1417724299 wait complete. 2024-02-19T22:21:18.063 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:17.231294+0000 mgr.smithi111.ftjzov (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:17.303386+0000 mon.smithi111 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:21:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:17.305193+0000 mon.smithi111 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:21:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:18.059312+0000 mon.smithi111 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.203:0/1417724299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:19.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:19.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:19.231628+0000 mgr.smithi111.ftjzov (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:21.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 -- 172.21.15.203:0/1104048651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 msgr2=0x7f68940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:21.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 --2- 172.21.15.203:0/1104048651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68940f55c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f6888009fb0 tx=0x7f6888009b30).stop 2024-02-19T22:21:21.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 -- 172.21.15.203:0/1104048651 shutdown_connections 2024-02-19T22:21:21.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 --2- 172.21.15.203:0/1104048651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68940f55c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:21.553 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 -- 172.21.15.203:0/1104048651 >> 172.21.15.203:0/1104048651 conn(0x7f68940f09a0 msgr2=0x7f68940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:21.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 -- 172.21.15.203:0/1104048651 shutdown_connections 2024-02-19T22:21:21.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.550+0000 7f6899d88700 1 -- 172.21.15.203:0/1104048651 wait complete. 2024-02-19T22:21:21.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6899d88700 1 Processor -- start 2024-02-19T22:21:21.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6899d88700 1 -- start start 2024-02-19T22:21:21.556 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6899d88700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6899d88700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941059c0 con 0x7f68940f51a0 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6898d86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6898d86700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41418/0 (socket says 172.21.15.203:41418) 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.551+0000 7f6898d86700 1 -- 172.21.15.203:0/3988733754 learned_addr learned my addr 172.21.15.203:0/3988733754 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f6898d86700 1 -- 172.21.15.203:0/3988733754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6888005040 con 0x7f68940f51a0 2024-02-19T22:21:21.559 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f6898d86700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f68880089f0 tx=0x7f688800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6888009320 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6894105bc0 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6888013380 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6888014e20 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.552+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6894106050 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.553+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68880263a0 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.553+0000 7f68917fa700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 0x7f68840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.553+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6888040f60 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.553+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6894045ad0 con 0x7f68940f51a0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.553+0000 7f6893fff700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 0x7f68840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:21.560 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.554+0000 7f6893fff700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 0x7f68840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f68800096a0 tx=0x7f6880006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:21.561 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.558+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6888012920 con 0x7f68940f51a0 2024-02-19T22:21:21.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.762+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6894106d70 con 0x7f68940f51a0 2024-02-19T22:21:21.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.763+0000 7f68917fa700 1 -- 172.21.15.203:0/3988733754 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f688802b020 con 0x7f68940f51a0 2024-02-19T22:21:21.766 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:21.766 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 msgr2=0x7f68840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 0x7f68840323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f68800096a0 tx=0x7f6880006b40).stop 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 msgr2=0x7f68941054d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f68880089f0 tx=0x7f688800b040).stop 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 shutdown_connections 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f688402ff10 0x7f68840323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:21.767 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 --2- 172.21.15.203:0/3988733754 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f68940f51a0 0x7f68941054d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:21.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.764+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 >> 172.21.15.203:0/3988733754 conn(0x7f68940f09a0 msgr2=0x7f68940f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:21.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.765+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 shutdown_connections 2024-02-19T22:21:21.768 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:21.765+0000 7f6899d88700 1 -- 172.21.15.203:0/3988733754 wait complete. 2024-02-19T22:21:21.768 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:21.231966+0000 mgr.smithi111.ftjzov (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:21.764377+0000 mon.smithi111 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.203:0/3988733754' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:23.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:23.624 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:23.232385+0000 mgr.smithi111.ftjzov (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:25.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 -- 172.21.15.203:0/1944289531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 msgr2=0x7f20c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:25.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/1944289531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c80f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f20b0009fb0 tx=0x7f20b0009b30).stop 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 -- 172.21.15.203:0/1944289531 shutdown_connections 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/1944289531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c80f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 -- 172.21.15.203:0/1944289531 >> 172.21.15.203:0/1944289531 conn(0x7f20c80f1c70 msgr2=0x7f20c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 -- 172.21.15.203:0/1944289531 shutdown_connections 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.438+0000 7f20ccb8a700 1 -- 172.21.15.203:0/1944289531 wait complete. 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20ccb8a700 1 Processor -- start 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20ccb8a700 1 -- start start 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20ccb8a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20ccb8a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8106cd0 con 0x7f20c80f6510 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20c659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20c659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37588/0 (socket says 172.21.15.203:37588) 2024-02-19T22:21:25.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20c659c700 1 -- 172.21.15.203:0/2052203725 learned_addr learned my addr 172.21.15.203:0/2052203725 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20c659c700 1 -- 172.21.15.203:0/2052203725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b0005040 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20c659c700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f20b00089f0 tx=0x7f20b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b0009320 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20b0013380 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.439+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c8106ed0 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.440+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f20b001ed80 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.440+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c8107360 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.440+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f20b001e420 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.440+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c8045ad0 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.440+0000 7f20beffd700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 0x7f20b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.441+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f20b0040e70 con 0x7f20c80f6510 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.441+0000 7f20c5d9b700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 0x7f20b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:25.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.441+0000 7f20c5d9b700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 0x7f20b4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f20b80096a0 tx=0x7f20b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:25.447 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.445+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f20b0014940 con 0x7f20c80f6510 2024-02-19T22:21:25.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.649+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20c8108210 con 0x7f20c80f6510 2024-02-19T22:21:25.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.649+0000 7f20beffd700 1 -- 172.21.15.203:0/2052203725 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f20b000e2a0 con 0x7f20c80f6510 2024-02-19T22:21:25.652 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:25.652 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 msgr2=0x7f20b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 0x7f20b4032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f20b80096a0 tx=0x7f20b8006b40).stop 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 msgr2=0x7f20c81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f20b00089f0 tx=0x7f20b000b040).stop 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 shutdown_connections 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f20b402ff60 0x7f20b4032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 --2- 172.21.15.203:0/2052203725 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f20c80f6510 0x7f20c81067e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 >> 172.21.15.203:0/2052203725 conn(0x7f20c80f1c70 msgr2=0x7f20c80f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 shutdown_connections 2024-02-19T22:21:25.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:25.651+0000 7f20ccb8a700 1 -- 172.21.15.203:0/2052203725 wait complete. 2024-02-19T22:21:25.655 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:25.232668+0000 mgr.smithi111.ftjzov (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:25.650811+0000 mon.smithi111 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.203:0/2052203725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:27.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:27.368 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:27.233008+0000 mgr.smithi111.ftjzov (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:29.356 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 -- 172.21.15.203:0/996221663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 msgr2=0x7f6cd40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 --2- 172.21.15.203:0/996221663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd40f6930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6cc4009fb0 tx=0x7f6cc4009b30).stop 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 -- 172.21.15.203:0/996221663 shutdown_connections 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 --2- 172.21.15.203:0/996221663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd40f6930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 -- 172.21.15.203:0/996221663 >> 172.21.15.203:0/996221663 conn(0x7f6cd40f1c70 msgr2=0x7f6cd40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 -- 172.21.15.203:0/996221663 shutdown_connections 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.354+0000 7f6cda785700 1 -- 172.21.15.203:0/996221663 wait complete. 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.355+0000 7f6cda785700 1 Processor -- start 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.355+0000 7f6cda785700 1 -- start start 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.355+0000 7f6cda785700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:29.357 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.355+0000 7f6cda785700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd4106cd0 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37594/0 (socket says 172.21.15.203:37594) 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd3fff700 1 -- 172.21.15.203:0/4065576301 learned_addr learned my addr 172.21.15.203:0/4065576301 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd3fff700 1 -- 172.21.15.203:0/4065576301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4005040 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd3fff700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6cc4005010 tx=0x7f6cc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc4009320 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd4106ed0 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6cc4013380 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cc4014e50 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.356+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd4107360 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.357+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6cc40095f0 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.357+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd4045ad0 con 0x7f6cd40f6510 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.357+0000 7f6cd0ff9700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 0x7f6cbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:29.359 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.357+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6cc4040ef0 con 0x7f6cd40f6510 2024-02-19T22:21:29.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.357+0000 7f6cd37fe700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 0x7f6cbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:29.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.358+0000 7f6cd37fe700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 0x7f6cbc0323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:29.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.362+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6cc4014420 con 0x7f6cd40f6510 2024-02-19T22:21:29.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.567+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cd41081b0 con 0x7f6cd40f6510 2024-02-19T22:21:29.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.567+0000 7f6cd0ff9700 1 -- 172.21.15.203:0/4065576301 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6cc4010230 con 0x7f6cd40f6510 2024-02-19T22:21:29.569 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:29.569 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 msgr2=0x7f6cbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 0x7f6cbc0323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).stop 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 msgr2=0x7f6cd41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6cc4005010 tx=0x7f6cc400b040).stop 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 shutdown_connections 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f6cbc02ff10 0x7f6cbc0323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 --2- 172.21.15.203:0/4065576301 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f6cd40f6510 0x7f6cd41067e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 >> 172.21.15.203:0/4065576301 conn(0x7f6cd40f1c70 msgr2=0x7f6cd40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 shutdown_connections 2024-02-19T22:21:29.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:29.569+0000 7f6cda785700 1 -- 172.21.15.203:0/4065576301 wait complete. 2024-02-19T22:21:29.572 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:29.233449+0000 mgr.smithi111.ftjzov (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:30.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:29.567882+0000 mon.smithi111 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.203:0/4065576301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:31.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:31.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:31.233696+0000 mgr.smithi111.ftjzov (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/2447409924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 msgr2=0x7f0db40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/2447409924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f0da8009fb0 tx=0x7f0da8009b30).stop 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/2447409924 shutdown_connections 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/2447409924 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/2447409924 >> 172.21.15.203:0/2447409924 conn(0x7f0db40f0b70 msgr2=0x7f0db40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/2447409924 shutdown_connections 2024-02-19T22:21:33.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/2447409924 wait complete. 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 Processor -- start 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.331+0000 7f0dbbb3e700 1 -- start start 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0dbbb3e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0dbbb3e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db40fb960 con 0x7f0db40f5410 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0db98da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0db98da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37620/0 (socket says 172.21.15.203:37620) 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0db98da700 1 -- 172.21.15.203:0/4033198777 learned_addr learned my addr 172.21.15.203:0/4033198777 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0db98da700 1 -- 172.21.15.203:0/4033198777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da8005040 con 0x7f0db40f5410 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0db98da700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f0da80089f0 tx=0x7f0da800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da8009320 con 0x7f0db40f5410 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db40fbb00 con 0x7f0db40f5410 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db40fbf90 con 0x7f0db40f5410 2024-02-19T22:21:33.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0da8013380 con 0x7f0db40f5410 2024-02-19T22:21:33.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.332+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0da801ed80 con 0x7f0db40f5410 2024-02-19T22:21:33.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.333+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0da801e420 con 0x7f0db40f5410 2024-02-19T22:21:33.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.333+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c004fa0 con 0x7f0db40f5410 2024-02-19T22:21:33.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.333+0000 7f0da67fc700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 0x7f0da0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:33.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.334+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0da8040ec0 con 0x7f0db40f5410 2024-02-19T22:21:33.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.334+0000 7f0db90d9700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 0x7f0da0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:33.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.334+0000 7f0db90d9700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 0x7f0da0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0db00096a0 tx=0x7f0db0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:33.339 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.338+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0da80095f0 con 0x7f0db40f5410 2024-02-19T22:21:33.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.542+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d9c005670 con 0x7f0db40f5410 2024-02-19T22:21:33.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.542+0000 7f0da67fc700 1 -- 172.21.15.203:0/4033198777 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0da801cb10 con 0x7f0db40f5410 2024-02-19T22:21:33.544 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:33.544 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 msgr2=0x7f0da0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 0x7f0da0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f0db00096a0 tx=0x7f0db0006b40).stop 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 msgr2=0x7f0db40fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f0da80089f0 tx=0x7f0da800b040).stop 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 shutdown_connections 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f0da002ff60 0x7f0da0032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 --2- 172.21.15.203:0/4033198777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f0db40f5410 0x7f0db40fd4e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 >> 172.21.15.203:0/4033198777 conn(0x7f0db40f0b70 msgr2=0x7f0db405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 shutdown_connections 2024-02-19T22:21:33.546 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:33.544+0000 7f0dbbb3e700 1 -- 172.21.15.203:0/4033198777 wait complete. 2024-02-19T22:21:33.547 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:33.233955+0000 mgr.smithi111.ftjzov (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:33.542821+0000 mon.smithi111 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.203:0/4033198777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:35.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:35.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:35.234333+0000 mgr.smithi111.ftjzov (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:37.186 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 -- 172.21.15.203:0/2973323349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 msgr2=0x7f26a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 --2- 172.21.15.203:0/2973323349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a00f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f2688009fb0 tx=0x7f2688009b30).stop 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 -- 172.21.15.203:0/2973323349 shutdown_connections 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 --2- 172.21.15.203:0/2973323349 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a00f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 -- 172.21.15.203:0/2973323349 >> 172.21.15.203:0/2973323349 conn(0x7f26a00f1c70 msgr2=0x7f26a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 -- 172.21.15.203:0/2973323349 shutdown_connections 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.182+0000 7f26a52e4700 1 -- 172.21.15.203:0/2973323349 wait complete. 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f26a52e4700 1 Processor -- start 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f26a52e4700 1 -- start start 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f26a52e4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:37.187 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f26a52e4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0106cd0 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f269effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f269effd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41270/0 (socket says 172.21.15.203:41270) 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.183+0000 7f269effd700 1 -- 172.21.15.203:0/3234031516 learned_addr learned my addr 172.21.15.203:0/3234031516 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f269effd700 1 -- 172.21.15.203:0/3234031516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2688005040 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f269effd700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2688005010 tx=0x7f268800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2688009320 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a0106ed0 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2688013380 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2688014e50 con 0x7f26a00f6510 2024-02-19T22:21:37.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.184+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a0107360 con 0x7f26a00f6510 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.185+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2688014420 con 0x7f26a00f6510 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.185+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0045ad0 con 0x7f26a00f6510 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.185+0000 7f26977fe700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 0x7f268c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.185+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2688040e30 con 0x7f26a00f6510 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.185+0000 7f269e7fc700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 0x7f268c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:37.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.186+0000 7f269e7fc700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 0x7f268c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f26900096a0 tx=0x7f2690006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:37.192 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.190+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26880095f0 con 0x7f26a00f6510 2024-02-19T22:21:37.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.396+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26a0108210 con 0x7f26a00f6510 2024-02-19T22:21:37.399 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.397+0000 7f26977fe700 1 -- 172.21.15.203:0/3234031516 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f268801db70 con 0x7f26a00f6510 2024-02-19T22:21:37.400 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:37.400 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.398+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 msgr2=0x7f268c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.398+0000 7f26a52e4700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 0x7f268c0323d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f26900096a0 tx=0x7f2690006b40).stop 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.398+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 msgr2=0x7f26a01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.398+0000 7f26a52e4700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f2688005010 tx=0x7f268800b040).stop 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 shutdown_connections 2024-02-19T22:21:37.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f268c02ff10 0x7f268c0323d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:37.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 --2- 172.21.15.203:0/3234031516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26a00f6510 0x7f26a01067e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:37.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 >> 172.21.15.203:0/3234031516 conn(0x7f26a00f1c70 msgr2=0x7f26a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:37.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 shutdown_connections 2024-02-19T22:21:37.402 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:37.399+0000 7f26a52e4700 1 -- 172.21.15.203:0/3234031516 wait complete. 2024-02-19T22:21:37.402 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:37.234708+0000 mgr.smithi111.ftjzov (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:37.398424+0000 mon.smithi111 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.203:0/3234031516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:39.092 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:39.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:39.235088+0000 mgr.smithi111.ftjzov (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.054+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/3894203065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 msgr2=0x7fdfb40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.054+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/3894203065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb40f6930 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdfa4009fb0 tx=0x7fdfa4009b30).stop 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.054+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/3894203065 shutdown_connections 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.054+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/3894203065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb40f6930 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.054+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/3894203065 >> 172.21.15.203:0/3894203065 conn(0x7fdfb40f1c70 msgr2=0x7fdfb40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/3894203065 shutdown_connections 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/3894203065 wait complete. 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 Processor -- start 2024-02-19T22:21:41.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 -- start start 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfbb4f6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb4108eb0 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfb9292700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfb9292700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41284/0 (socket says 172.21.15.203:41284) 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.055+0000 7fdfb9292700 1 -- 172.21.15.203:0/618756810 learned_addr learned my addr 172.21.15.203:0/618756810 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfb9292700 1 -- 172.21.15.203:0/618756810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4005040 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfb9292700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fdfa4005010 tx=0x7fdfa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4009320 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb41090b0 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdfa4013380 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4014e50 con 0x7fdfb40f6510 2024-02-19T22:21:41.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.056+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb4109540 con 0x7fdfb40f6510 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.057+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfa40095f0 con 0x7fdfb40f6510 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.057+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb4045ad0 con 0x7fdfb40f6510 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.057+0000 7fdfa9ffb700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.057+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfa4040e70 con 0x7fdfb40f6510 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.057+0000 7fdfb8a91700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:41.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.058+0000 7fdfb8a91700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 0x7fdfa00323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:41.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.062+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa4014420 con 0x7fdfb40f6510 2024-02-19T22:21:41.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.266+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfb410a3f0 con 0x7fdfb40f6510 2024-02-19T22:21:41.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.266+0000 7fdfa9ffb700 1 -- 172.21.15.203:0/618756810 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfa4010230 con 0x7fdfb40f6510 2024-02-19T22:21:41.269 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:41.270 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 msgr2=0x7fdfa00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 0x7fdfa00323d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fdfb00096a0 tx=0x7fdfb0006b40).stop 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 msgr2=0x7fdfb41089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fdfa4005010 tx=0x7fdfa400b040).stop 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 shutdown_connections 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fdfa002ff10 0x7fdfa00323d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 --2- 172.21.15.203:0/618756810 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fdfb40f6510 0x7fdfb41089c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 >> 172.21.15.203:0/618756810 conn(0x7fdfb40f1c70 msgr2=0x7fdfb40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 shutdown_connections 2024-02-19T22:21:41.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:41.268+0000 7fdfbb4f6700 1 -- 172.21.15.203:0/618756810 wait complete. 2024-02-19T22:21:41.272 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:42.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:41.235331+0000 mgr.smithi111.ftjzov (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:42.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:41.268022+0000 mon.smithi111 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.203:0/618756810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:43.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:43.040 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:43.235628+0000 mgr.smithi111.ftjzov (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 -- 172.21.15.203:0/3312799056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 msgr2=0x7fc7e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/3312799056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e00f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fc7d4009fb0 tx=0x7fc7d4009b30).stop 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 -- 172.21.15.203:0/3312799056 shutdown_connections 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/3312799056 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e00f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 -- 172.21.15.203:0/3312799056 >> 172.21.15.203:0/3312799056 conn(0x7fc7e00f0b70 msgr2=0x7fc7e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 -- 172.21.15.203:0/3312799056 shutdown_connections 2024-02-19T22:21:44.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.950+0000 7fc7e5a49700 1 -- 172.21.15.203:0/3312799056 wait complete. 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.951+0000 7fc7e5a49700 1 Processor -- start 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.951+0000 7fc7e5a49700 1 -- start start 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.951+0000 7fc7e5a49700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.951+0000 7fc7e5a49700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0106cd0 con 0x7fc7e00f5410 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7deffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7deffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41308/0 (socket says 172.21.15.203:41308) 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7deffd700 1 -- 172.21.15.203:0/19366130 learned_addr learned my addr 172.21.15.203:0/19366130 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7deffd700 1 -- 172.21.15.203:0/19366130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4005040 con 0x7fc7e00f5410 2024-02-19T22:21:44.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7deffd700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc7d4005010 tx=0x7fc7d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d4009320 con 0x7fc7e00f5410 2024-02-19T22:21:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc7d4013380 con 0x7fc7e00f5410 2024-02-19T22:21:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e0106ed0 con 0x7fc7e00f5410 2024-02-19T22:21:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7d4014e50 con 0x7fc7e00f5410 2024-02-19T22:21:44.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.952+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e0107360 con 0x7fc7e00f5410 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.953+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7d40263a0 con 0x7fc7e00f5410 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.953+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e0045ad0 con 0x7fc7e00f5410 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.953+0000 7fc7c77fe700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 0x7fc7c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.953+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7d4041270 con 0x7fc7e00f5410 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.954+0000 7fc7de7fc700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 0x7fc7c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:44.957 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.954+0000 7fc7de7fc700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 0x7fc7c80323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc7d00096a0 tx=0x7fc7d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:44.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:44.958+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7d40126f0 con 0x7fc7e00f5410 2024-02-19T22:21:45.166 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.163+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7e0108210 con 0x7fc7e00f5410 2024-02-19T22:21:45.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.164+0000 7fc7c77fe700 1 -- 172.21.15.203:0/19366130 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc7d4012330 con 0x7fc7e00f5410 2024-02-19T22:21:45.167 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:45.167 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:45.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 msgr2=0x7fc7c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:45.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 0x7fc7c80323d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fc7d00096a0 tx=0x7fc7d0006b40).stop 2024-02-19T22:21:45.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 msgr2=0x7fc7e01067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:45.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fc7d4005010 tx=0x7fc7d400b040).stop 2024-02-19T22:21:45.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 shutdown_connections 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc7c802ff10 0x7fc7c80323d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 --2- 172.21.15.203:0/19366130 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc7e00f5410 0x7fc7e01067e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 >> 172.21.15.203:0/19366130 conn(0x7fc7e00f0b70 msgr2=0x7fc7e00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 shutdown_connections 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:45.166+0000 7fc7e5a49700 1 -- 172.21.15.203:0/19366130 wait complete. 2024-02-19T22:21:45.169 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:45.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:45.165585+0000 mon.smithi111 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.203:0/19366130' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:46.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:45.235889+0000 mgr.smithi111.ftjzov (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:46.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:46.861 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:47.236267+0000 mgr.smithi111.ftjzov (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 -- 172.21.15.203:0/604433395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 msgr2=0x7f03b00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/604433395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b00f5830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f03a0009fb0 tx=0x7f03a0009b30).stop 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 -- 172.21.15.203:0/604433395 shutdown_connections 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/604433395 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b00f5830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 -- 172.21.15.203:0/604433395 >> 172.21.15.203:0/604433395 conn(0x7f03b00f0b70 msgr2=0x7f03b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 -- 172.21.15.203:0/604433395 shutdown_connections 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.818+0000 7f03b7b6a700 1 -- 172.21.15.203:0/604433395 wait complete. 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b7b6a700 1 Processor -- start 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b7b6a700 1 -- start start 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b7b6a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b7b6a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0106cc0 con 0x7f03b00f5410 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b5906700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:48.823 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b5906700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41410/0 (socket says 172.21.15.203:41410) 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b5906700 1 -- 172.21.15.203:0/2703474307 learned_addr learned my addr 172.21.15.203:0/2703474307 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b5906700 1 -- 172.21.15.203:0/2703474307 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0005040 con 0x7f03b00f5410 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.819+0000 7f03b5906700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f03a0005010 tx=0x7f03a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a0009320 con 0x7f03b00f5410 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b0106ec0 con 0x7f03b00f5410 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a0013380 con 0x7f03b00f5410 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a0014e50 con 0x7f03b00f5410 2024-02-19T22:21:48.824 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b0107350 con 0x7f03b00f5410 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f03a0014420 con 0x7f03b00f5410 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.820+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b0045ad0 con 0x7f03b00f5410 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.821+0000 7f03a67fc700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 0x7f039c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.821+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f03a0040f40 con 0x7f03b00f5410 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.821+0000 7f03b5105700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 0x7f039c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:48.825 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.821+0000 7f03b5105700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 0x7f039c0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:48.828 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:48.825+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f03a00095f0 con 0x7f03b00f5410 2024-02-19T22:21:49.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.031+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03b0108200 con 0x7f03b00f5410 2024-02-19T22:21:49.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.031+0000 7f03a67fc700 1 -- 172.21.15.203:0/2703474307 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f03a001db70 con 0x7f03b00f5410 2024-02-19T22:21:49.034 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:49.034 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 msgr2=0x7f039c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 0x7f039c0323d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f03ac0096a0 tx=0x7f03ac006b40).stop 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 msgr2=0x7f03b01067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f03a0005010 tx=0x7f03a000b040).stop 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 shutdown_connections 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f039c02ff10 0x7f039c0323d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 --2- 172.21.15.203:0/2703474307 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f03b00f5410 0x7f03b01067d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 >> 172.21.15.203:0/2703474307 conn(0x7f03b00f0b70 msgr2=0x7f03b005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 shutdown_connections 2024-02-19T22:21:49.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:49.033+0000 7f03b7b6a700 1 -- 172.21.15.203:0/2703474307 wait complete. 2024-02-19T22:21:49.037 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:49.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:49 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:49.032945+0000 mon.smithi111 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.203:0/2703474307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:49.236579+0000 mgr.smithi111.ftjzov (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:50.798 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:50.799 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 -- 172.21.15.203:0/2151932228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 msgr2=0x7fd7cc0f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2151932228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc0f91b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fd7c0009fb0 tx=0x7fd7c0009b30).stop 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 -- 172.21.15.203:0/2151932228 shutdown_connections 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2151932228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc0f91b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 -- 172.21.15.203:0/2151932228 >> 172.21.15.203:0/2151932228 conn(0x7fd7cc0f0b70 msgr2=0x7fd7cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 -- 172.21.15.203:0/2151932228 shutdown_connections 2024-02-19T22:21:52.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.693+0000 7fd7d2653700 1 -- 172.21.15.203:0/2151932228 wait complete. 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.694+0000 7fd7d2653700 1 Processor -- start 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.694+0000 7fd7d2653700 1 -- start start 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.694+0000 7fd7d2653700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.694+0000 7fd7d2653700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc106cb0 con 0x7fd7cc0601c0 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41434/0 (socket says 172.21.15.203:41434) 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7cbfff700 1 -- 172.21.15.203:0/2508598403 learned_addr learned my addr 172.21.15.203:0/2508598403 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7cbfff700 1 -- 172.21.15.203:0/2508598403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0005040 con 0x7fd7cc0601c0 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7cbfff700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd7c0009830 tx=0x7fd7c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:52.700 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c0009320 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc106eb0 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7c0013380 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7c0014420 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.695+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc107340 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.696+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7c0014580 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.696+0000 7fd7c8ff9700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 0x7fd7b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.696+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc045ad0 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.696+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7c0040df0 con 0x7fd7cc0601c0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.696+0000 7fd7cb7fe700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 0x7fd7b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:52.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.697+0000 7fd7cb7fe700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 0x7fd7b4032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd7bc0096a0 tx=0x7fd7bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:52.703 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.701+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7c00273f0 con 0x7fd7cc0601c0 2024-02-19T22:21:52.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:51.236877+0000 mgr.smithi111.ftjzov (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:52.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.904+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7cc0fc090 con 0x7fd7cc0601c0 2024-02-19T22:21:52.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.905+0000 7fd7c8ff9700 1 -- 172.21.15.203:0/2508598403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd7c001c890 con 0x7fd7cc0601c0 2024-02-19T22:21:52.907 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:52.907 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 msgr2=0x7fd7b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 0x7fd7b4032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fd7bc0096a0 tx=0x7fd7bc006b40).stop 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 msgr2=0x7fd7cc1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fd7c0009830 tx=0x7fd7c000b040).stop 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 shutdown_connections 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd7b402ff60 0x7fd7b4032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 --2- 172.21.15.203:0/2508598403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7cc0601c0 0x7fd7cc1067c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 >> 172.21.15.203:0/2508598403 conn(0x7fd7cc0f0b70 msgr2=0x7fd7cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:52.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 shutdown_connections 2024-02-19T22:21:52.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:52.906+0000 7fd7d2653700 1 -- 172.21.15.203:0/2508598403 wait complete. 2024-02-19T22:21:52.910 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:53.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:52.906157+0000 mon.smithi111 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.203:0/2508598403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:54.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:54.616 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:53.237147+0000 mgr.smithi111.ftjzov (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:56.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 -- 172.21.15.203:0/106354944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 msgr2=0x7f8fd00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:56.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 --2- 172.21.15.203:0/106354944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd00f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f8fc4009fb0 tx=0x7f8fc4009b30).stop 2024-02-19T22:21:56.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 -- 172.21.15.203:0/106354944 shutdown_connections 2024-02-19T22:21:56.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 --2- 172.21.15.203:0/106354944 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd00f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:56.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 -- 172.21.15.203:0/106354944 >> 172.21.15.203:0/106354944 conn(0x7f8fd00f1c70 msgr2=0x7f8fd00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 -- 172.21.15.203:0/106354944 shutdown_connections 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.615+0000 7f8fd5271700 1 -- 172.21.15.203:0/106354944 wait complete. 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fd5271700 1 Processor -- start 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fd5271700 1 -- start start 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fd5271700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fd5271700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd0069e20 con 0x7f8fd00f6510 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fceffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fceffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34542/0 (socket says 172.21.15.203:34542) 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.616+0000 7f8fceffd700 1 -- 172.21.15.203:0/4091569516 learned_addr learned my addr 172.21.15.203:0/4091569516 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fceffd700 1 -- 172.21.15.203:0/4091569516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4005040 con 0x7f8fd00f6510 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fceffd700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f8fc4005010 tx=0x7f8fc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fc4009320 con 0x7f8fd00f6510 2024-02-19T22:21:56.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd0069fc0 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fc4013380 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8fc4014e50 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.617+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd006a450 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.618+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8fc4014420 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.618+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd0059100 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.618+0000 7f8fb77fe700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 0x7f8fb80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.618+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8fc4040e30 con 0x7f8fd00f6510 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.618+0000 7f8fce7fc700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 0x7f8fb80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:21:56.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.619+0000 7f8fce7fc700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 0x7f8fb80323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8fc00096a0 tx=0x7f8fc0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:21:56.625 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.623+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8fc40095f0 con 0x7f8fd00f6510 2024-02-19T22:21:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:55.237433+0000 mgr.smithi111.ftjzov (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:21:56.829 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.827+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8fd00fc0a0 con 0x7f8fd00f6510 2024-02-19T22:21:56.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.827+0000 7f8fb77fe700 1 -- 172.21.15.203:0/4091569516 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8fc401db70 con 0x7f8fd00f6510 2024-02-19T22:21:56.830 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:21:56.830 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:21:56.831 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 msgr2=0x7f8fb80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 0x7f8fb80323d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f8fc00096a0 tx=0x7f8fc0006b40).stop 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 msgr2=0x7f8fd006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f8fc4005010 tx=0x7f8fc400b040).stop 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 shutdown_connections 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f8fb802ff10 0x7f8fb80323d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 --2- 172.21.15.203:0/4091569516 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f8fd00f6510 0x7f8fd006b9a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 >> 172.21.15.203:0/4091569516 conn(0x7f8fd00f1c70 msgr2=0x7f8fd00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 shutdown_connections 2024-02-19T22:21:56.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:21:56.829+0000 7f8fd5271700 1 -- 172.21.15.203:0/4091569516 wait complete. 2024-02-19T22:21:56.833 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:21:57.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:21:56.828672+0000 mon.smithi111 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.203:0/4091569516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:21:58.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:21:58.507 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:21:58.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:21:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:57.237715+0000 mgr.smithi111.ftjzov (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- 172.21.15.203:0/2946619642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 msgr2=0x7ff4740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/2946619642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4740f5830 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7ff464009fb0 tx=0x7ff464009b30).stop 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- 172.21.15.203:0/2946619642 shutdown_connections 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/2946619642 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4740f5830 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- 172.21.15.203:0/2946619642 >> 172.21.15.203:0/2946619642 conn(0x7ff4740f0b70 msgr2=0x7ff4740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:00.433 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- 172.21.15.203:0/2946619642 shutdown_connections 2024-02-19T22:22:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- 172.21.15.203:0/2946619642 wait complete. 2024-02-19T22:22:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 Processor -- start 2024-02-19T22:22:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.430+0000 7ff47bdcd700 1 -- start start 2024-02-19T22:22:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff47bdcd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:00.434 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff47bdcd700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff474106ca0 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff479b69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff479b69700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34562/0 (socket says 172.21.15.203:34562) 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff479b69700 1 -- 172.21.15.203:0/1482593104 learned_addr learned my addr 172.21.15.203:0/1482593104 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff479b69700 1 -- 172.21.15.203:0/1482593104 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff464005040 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff479b69700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7ff464005010 tx=0x7ff46400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.431+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff464009320 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff474106ea0 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff464013380 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff464014e50 con 0x7ff4740f5410 2024-02-19T22:22:00.436 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff474107330 con 0x7ff4740f5410 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4640095f0 con 0x7ff4740f5410 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff474045ad0 con 0x7ff4740f5410 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff46a7fc700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 0x7ff4600323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.432+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff464040f80 con 0x7ff4740f5410 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.433+0000 7ff479368700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 0x7ff4600323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:00.437 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.433+0000 7ff479368700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 0x7ff4600323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff4700096a0 tx=0x7ff470006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:00.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.437+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff464014420 con 0x7ff4740f5410 2024-02-19T22:22:00.646 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.644+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4740fafa0 con 0x7ff4740f5410 2024-02-19T22:22:00.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.644+0000 7ff46a7fc700 1 -- 172.21.15.203:0/1482593104 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff464010230 con 0x7ff4740f5410 2024-02-19T22:22:00.647 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:00.647 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 msgr2=0x7ff4600323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 0x7ff4600323d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7ff4700096a0 tx=0x7ff470006b40).stop 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 msgr2=0x7ff4741067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7ff464005010 tx=0x7ff46400b040).stop 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 shutdown_connections 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff46002ff10 0x7ff4600323d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 --2- 172.21.15.203:0/1482593104 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff4740f5410 0x7ff4741067b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 >> 172.21.15.203:0/1482593104 conn(0x7ff4740f0b70 msgr2=0x7ff47405fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 shutdown_connections 2024-02-19T22:22:00.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:00.646+0000 7ff47bdcd700 1 -- 172.21.15.203:0/1482593104 wait complete. 2024-02-19T22:22:00.650 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:21:59.238081+0000 mgr.smithi111.ftjzov (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:01.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:00.645785+0000 mon.smithi111 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.203:0/1482593104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:02.408 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:02.408 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:02.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:01.238421+0000 mgr.smithi111.ftjzov (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:04.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 -- 172.21.15.203:0/4131472861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 msgr2=0x7f9d000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 --2- 172.21.15.203:0/4131472861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d000f6930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9ce8009fb0 tx=0x7f9ce8009b30).stop 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 -- 172.21.15.203:0/4131472861 shutdown_connections 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 --2- 172.21.15.203:0/4131472861 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d000f6930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 -- 172.21.15.203:0/4131472861 >> 172.21.15.203:0/4131472861 conn(0x7f9d000f1c70 msgr2=0x7f9d000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 -- 172.21.15.203:0/4131472861 shutdown_connections 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.355+0000 7f9d054ca700 1 -- 172.21.15.203:0/4131472861 wait complete. 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9d054ca700 1 Processor -- start 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9d054ca700 1 -- start start 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9d054ca700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:04.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9d054ca700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00106cd0 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9cfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9cfeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:34578/0 (socket says 172.21.15.203:34578) 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.356+0000 7f9cfeffd700 1 -- 172.21.15.203:0/2691912510 learned_addr learned my addr 172.21.15.203:0/2691912510 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9cfeffd700 1 -- 172.21.15.203:0/2691912510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8005040 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9cfeffd700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f9ce8005010 tx=0x7f9ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce8009320 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ce8013380 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d00106ed0 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ce8014e50 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.357+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00107360 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.358+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ce80095f0 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.358+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d00045ad0 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.358+0000 7f9cf77fe700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 0x7f9cec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.358+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ce8040f80 con 0x7f9d000f6510 2024-02-19T22:22:04.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.358+0000 7f9cfe7fc700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 0x7f9cec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:04.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.359+0000 7f9cfe7fc700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 0x7f9cec0323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9cf00096a0 tx=0x7f9cf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:04.368 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.363+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ce8014420 con 0x7f9d000f6510 2024-02-19T22:22:04.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.569+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d00108210 con 0x7f9d000f6510 2024-02-19T22:22:04.571 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.569+0000 7f9cf77fe700 1 -- 172.21.15.203:0/2691912510 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ce8010230 con 0x7f9d000f6510 2024-02-19T22:22:04.572 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:04.572 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 msgr2=0x7f9cec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 0x7f9cec0323d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9cf00096a0 tx=0x7f9cf0006b40).stop 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 msgr2=0x7f9d001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f9ce8005010 tx=0x7f9ce800b040).stop 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 shutdown_connections 2024-02-19T22:22:04.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f9cec02ff10 0x7f9cec0323d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:04.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 --2- 172.21.15.203:0/2691912510 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f9d000f6510 0x7f9d001067e0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:04.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 >> 172.21.15.203:0/2691912510 conn(0x7f9d000f1c70 msgr2=0x7f9d000f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:04.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 shutdown_connections 2024-02-19T22:22:04.574 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:04.571+0000 7f9d054ca700 1 -- 172.21.15.203:0/2691912510 wait complete. 2024-02-19T22:22:04.574 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:03.238712+0000 mgr.smithi111.ftjzov (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:05.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:04.570810+0000 mon.smithi111 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.203:0/2691912510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:06.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:06.279 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:05.239026+0000 mgr.smithi111.ftjzov (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:08.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 -- 172.21.15.203:0/3056849208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 msgr2=0x7f49040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:08.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 --2- 172.21.15.203:0/3056849208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f49040f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f48f4009fb0 tx=0x7f48f4009b30).stop 2024-02-19T22:22:08.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 -- 172.21.15.203:0/3056849208 shutdown_connections 2024-02-19T22:22:08.188 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 --2- 172.21.15.203:0/3056849208 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f49040f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 -- 172.21.15.203:0/3056849208 >> 172.21.15.203:0/3056849208 conn(0x7f49040f0b70 msgr2=0x7f49040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 -- 172.21.15.203:0/3056849208 shutdown_connections 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.166+0000 7f490c666700 1 -- 172.21.15.203:0/3056849208 wait complete. 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490c666700 1 Processor -- start 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490c666700 1 -- start start 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490c666700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490c666700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4904060220 con 0x7f49040f5410 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490a402700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490a402700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38200/0 (socket says 172.21.15.203:38200) 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490a402700 1 -- 172.21.15.203:0/3056373648 learned_addr learned my addr 172.21.15.203:0/3056373648 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.167+0000 7f490a402700 1 -- 172.21.15.203:0/3056373648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f4005040 con 0x7f49040f5410 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f490a402700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f48f4005010 tx=0x7f48f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f4009320 con 0x7f49040f5410 2024-02-19T22:22:08.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4904060420 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48f4013380 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48f4014e50 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.168+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4904109bf0 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48f40095f0 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f48faffd700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 0x7f48f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904059100 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48f4040f30 con 0x7f49040f5410 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f4909c01700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 0x7f48f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.169+0000 7f4909c01700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 0x7f48f00323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f49000096a0 tx=0x7f4900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:08.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.174+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48f401db60 con 0x7f49040f5410 2024-02-19T22:22:08.379 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.378+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49040fafa0 con 0x7f49040f5410 2024-02-19T22:22:08.380 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.378+0000 7f48faffd700 1 -- 172.21.15.203:0/3056373648 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f48f401dd70 con 0x7f49040f5410 2024-02-19T22:22:08.380 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:08.380 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 msgr2=0x7f48f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 0x7f48f00323d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f49000096a0 tx=0x7f4900006b40).stop 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 msgr2=0x7f4904109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f48f4005010 tx=0x7f48f400b040).stop 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 shutdown_connections 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f48f002ff10 0x7f48f00323d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 --2- 172.21.15.203:0/3056373648 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f49040f5410 0x7f4904109540 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 >> 172.21.15.203:0/3056373648 conn(0x7f49040f0b70 msgr2=0x7f49040f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 shutdown_connections 2024-02-19T22:22:08.382 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:08.380+0000 7f490c666700 1 -- 172.21.15.203:0/3056373648 wait complete. 2024-02-19T22:22:08.383 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:07.239434+0000 mgr.smithi111.ftjzov (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:09.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:09 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:08.378613+0000 mon.smithi111 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.203:0/3056373648' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:10.077 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:10.077 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:09.239720+0000 mgr.smithi111.ftjzov (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:12.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 -- 172.21.15.203:0/1257957052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 msgr2=0x7f7b580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:12.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 --2- 172.21.15.203:0/1257957052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b580f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f7b4c009fb0 tx=0x7f7b4c009b30).stop 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 -- 172.21.15.203:0/1257957052 shutdown_connections 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 --2- 172.21.15.203:0/1257957052 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b580f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 -- 172.21.15.203:0/1257957052 >> 172.21.15.203:0/1257957052 conn(0x7f7b580f0b70 msgr2=0x7f7b580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 -- 172.21.15.203:0/1257957052 shutdown_connections 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.057+0000 7f7b5df35700 1 -- 172.21.15.203:0/1257957052 wait complete. 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b5df35700 1 Processor -- start 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b5df35700 1 -- start start 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b5df35700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b5df35700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b58105bd0 con 0x7f7b580f5410 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b577fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:38224/0 (socket says 172.21.15.203:38224) 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.058+0000 7f7b577fe700 1 -- 172.21.15.203:0/536644011 learned_addr learned my addr 172.21.15.203:0/536644011 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b577fe700 1 -- 172.21.15.203:0/536644011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c005040 con 0x7f7b580f5410 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b577fe700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7b4c005010 tx=0x7f7b4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:12.081 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b4c009320 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b58105dd0 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b4c013380 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b4c014e50 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.059+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b58106260 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b4c0095f0 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58045ad0 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b3ffff700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 0x7f7b400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b4c040fa0 con 0x7f7b580f5410 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b56ffd700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 0x7f7b400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.060+0000 7f7b56ffd700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 0x7f7b400323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7b480096a0 tx=0x7f7b48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:12.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.064+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b4c014420 con 0x7f7b580f5410 2024-02-19T22:22:12.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.271+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b58107110 con 0x7f7b580f5410 2024-02-19T22:22:12.273 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.272+0000 7f7b3ffff700 1 -- 172.21.15.203:0/536644011 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7b4c010230 con 0x7f7b580f5410 2024-02-19T22:22:12.274 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:12.274 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:12.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 msgr2=0x7f7b400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:12.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 0x7f7b400323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f7b480096a0 tx=0x7f7b48006b40).stop 2024-02-19T22:22:12.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 msgr2=0x7f7b581056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:12.275 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f7b4c005010 tx=0x7f7b4c00b040).stop 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 shutdown_connections 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7b4002ff10 0x7f7b400323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 --2- 172.21.15.203:0/536644011 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7b580f5410 0x7f7b581056e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 >> 172.21.15.203:0/536644011 conn(0x7f7b580f0b70 msgr2=0x7f7b580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 shutdown_connections 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:12.274+0000 7f7b5df35700 1 -- 172.21.15.203:0/536644011 wait complete. 2024-02-19T22:22:12.276 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:11.239947+0000 mgr.smithi111.ftjzov (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:13.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:13 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:12.272331+0000 mon.smithi111 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.203:0/536644011' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:14.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:14.007 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:13.240296+0000 mgr.smithi111.ftjzov (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 -- 172.21.15.203:0/177093389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 msgr2=0x7fb6a80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/177093389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a80f5830 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7fb69c009fb0 tx=0x7fb69c009b30).stop 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 -- 172.21.15.203:0/177093389 shutdown_connections 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/177093389 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a80f5830 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 -- 172.21.15.203:0/177093389 >> 172.21.15.203:0/177093389 conn(0x7fb6a80f0b70 msgr2=0x7fb6a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 -- 172.21.15.203:0/177093389 shutdown_connections 2024-02-19T22:22:16.080 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.077+0000 7fb6aef0e700 1 -- 172.21.15.203:0/177093389 wait complete. 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6aef0e700 1 Processor -- start 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6aef0e700 1 -- start start 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6aef0e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6aef0e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a8105bd0 con 0x7fb6a80f5410 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6accaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6accaa700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46764/0 (socket says 172.21.15.203:46764) 2024-02-19T22:22:16.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.078+0000 7fb6accaa700 1 -- 172.21.15.203:0/2119865692 learned_addr learned my addr 172.21.15.203:0/2119865692 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6accaa700 1 -- 172.21.15.203:0/2119865692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c005040 con 0x7fb6a80f5410 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6accaa700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb69c005010 tx=0x7fb69c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69c009320 con 0x7fb6a80f5410 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a8105dd0 con 0x7fb6a80f5410 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb69c013380 con 0x7fb6a80f5410 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb69c014e50 con 0x7fb6a80f5410 2024-02-19T22:22:16.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.079+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a8106260 con 0x7fb6a80f5410 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.080+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb69c014420 con 0x7fb6a80f5410 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.080+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a8045ad0 con 0x7fb6a80f5410 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.080+0000 7fb6a57fa700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.080+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb69c040e30 con 0x7fb6a80f5410 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.080+0000 7fb6a7fff700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:16.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.081+0000 7fb6a7fff700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 0x7fb6980323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb6940096a0 tx=0x7fb694006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:16.087 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.085+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb69c0095f0 con 0x7fb6a80f5410 2024-02-19T22:22:16.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.289+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6a8107110 con 0x7fb6a80f5410 2024-02-19T22:22:16.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.290+0000 7fb6a57fa700 1 -- 172.21.15.203:0/2119865692 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb69c01db70 con 0x7fb6a80f5410 2024-02-19T22:22:16.293 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:16.293 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:16.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 msgr2=0x7fb6980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:16.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 0x7fb6980323d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb6940096a0 tx=0x7fb694006b40).stop 2024-02-19T22:22:16.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 msgr2=0x7fb6a81056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:16.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fb69c005010 tx=0x7fb69c00b040).stop 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 shutdown_connections 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb69802ff10 0x7fb6980323d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 --2- 172.21.15.203:0/2119865692 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb6a80f5410 0x7fb6a81056e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 >> 172.21.15.203:0/2119865692 conn(0x7fb6a80f0b70 msgr2=0x7fb6a80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 shutdown_connections 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:16.292+0000 7fb6aef0e700 1 -- 172.21.15.203:0/2119865692 wait complete. 2024-02-19T22:22:16.295 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:16.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:15.240656+0000 mgr.smithi111.ftjzov (mgr.14162) 277 : cluster [DBG] pgmap v263: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:17.696 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:16.291515+0000 mon.smithi111 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.203:0/2119865692' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:18.020 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:18.020 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:17.241061+0000 mgr.smithi111.ftjzov (mgr.14162) 278 : cluster [DBG] pgmap v264: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:17.304372+0000 mon.smithi111 (mon.0) 274 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:22:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:17.306035+0000 mon.smithi111 (mon.0) 275 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:22:20.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2708996332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 msgr2=0x7f7bd40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:20.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2708996332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd40f90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f7bc8009fb0 tx=0x7f7bc8009b30).stop 2024-02-19T22:22:20.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2708996332 shutdown_connections 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2708996332 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd40f90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2708996332 >> 172.21.15.203:0/2708996332 conn(0x7f7bd40f0b70 msgr2=0x7f7bd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2708996332 shutdown_connections 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.127+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2708996332 wait complete. 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd8d4c700 1 Processor -- start 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd8d4c700 1 -- start start 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd8d4c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd8d4c700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd4106c90 con 0x7f7bd4060280 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:20.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46788/0 (socket says 172.21.15.203:46788) 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.128+0000 7f7bd259c700 1 -- 172.21.15.203:0/2145781198 learned_addr learned my addr 172.21.15.203:0/2145781198 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bd259c700 1 -- 172.21.15.203:0/2145781198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8005040 con 0x7f7bd4060280 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bd259c700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f7bc80089f0 tx=0x7f7bc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc8009320 con 0x7f7bd4060280 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bc8013380 con 0x7f7bd4060280 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bd4106e90 con 0x7f7bd4060280 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7bc8014e20 con 0x7f7bd4060280 2024-02-19T22:22:20.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.129+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bd4107320 con 0x7f7bd4060280 2024-02-19T22:22:20.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7bc80095f0 con 0x7f7bd4060280 2024-02-19T22:22:20.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bd4045ad0 con 0x7f7bd4060280 2024-02-19T22:22:20.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bc2ffd700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 0x7f7bbc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:20.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7bc8040f10 con 0x7f7bd4060280 2024-02-19T22:22:20.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bd1d9b700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 0x7f7bbc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:20.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.130+0000 7f7bd1d9b700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 0x7f7bbc0323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f7bc40096a0 tx=0x7f7bc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:20.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.135+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7bc8014420 con 0x7f7bd4060280 2024-02-19T22:22:20.344 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.342+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7bd41081d0 con 0x7f7bd4060280 2024-02-19T22:22:20.345 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.342+0000 7f7bc2ffd700 1 -- 172.21.15.203:0/2145781198 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7bc801d980 con 0x7f7bd4060280 2024-02-19T22:22:20.345 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:20.345 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:20.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 msgr2=0x7f7bbc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:20.346 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 0x7f7bbc0323d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f7bc40096a0 tx=0x7f7bc4006b40).stop 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 msgr2=0x7f7bd41067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f7bc80089f0 tx=0x7f7bc800b040).stop 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 shutdown_connections 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7bbc02ff10 0x7f7bbc0323d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 --2- 172.21.15.203:0/2145781198 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7bd4060280 0x7f7bd41067a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 >> 172.21.15.203:0/2145781198 conn(0x7f7bd40f0b70 msgr2=0x7f7bd40f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 shutdown_connections 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:20.344+0000 7f7bd8d4c700 1 -- 172.21.15.203:0/2145781198 wait complete. 2024-02-19T22:22:20.347 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:20.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:19.241415+0000 mgr.smithi111.ftjzov (mgr.14162) 279 : cluster [DBG] pgmap v265: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:21.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:21 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:20.343781+0000 mon.smithi111 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.203:0/2145781198' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:22.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:22.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:21.241631+0000 mgr.smithi111.ftjzov (mgr.14162) 280 : cluster [DBG] pgmap v266: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:24.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- 172.21.15.203:0/824400195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 msgr2=0x7fe8a00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:24.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 --2- 172.21.15.203:0/824400195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe890009fb0 tx=0x7fe890009b30).stop 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- 172.21.15.203:0/824400195 shutdown_connections 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 --2- 172.21.15.203:0/824400195 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a00f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- 172.21.15.203:0/824400195 >> 172.21.15.203:0/824400195 conn(0x7fe8a00f09a0 msgr2=0x7fe8a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- 172.21.15.203:0/824400195 shutdown_connections 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- 172.21.15.203:0/824400195 wait complete. 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 Processor -- start 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.065+0000 7fe8a566a700 1 -- start start 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe8a566a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe8a566a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a0060280 con 0x7fe8a00f51a0 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89ffff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46798/0 (socket says 172.21.15.203:46798) 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89ffff700 1 -- 172.21.15.203:0/868807466 learned_addr learned my addr 172.21.15.203:0/868807466 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89ffff700 1 -- 172.21.15.203:0/868807466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe890005040 con 0x7fe8a00f51a0 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89ffff700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe890009830 tx=0x7fe89000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:24.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe890009320 con 0x7fe8a00f51a0 2024-02-19T22:22:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a0060480 con 0x7fe8a00f51a0 2024-02-19T22:22:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe890013380 con 0x7fe8a00f51a0 2024-02-19T22:22:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe890014420 con 0x7fe8a00f51a0 2024-02-19T22:22:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.066+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a01099c0 con 0x7fe8a00f51a0 2024-02-19T22:22:24.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.067+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0045ad0 con 0x7fe8a00f51a0 2024-02-19T22:22:24.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.067+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe89001e420 con 0x7fe8a00f51a0 2024-02-19T22:22:24.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.067+0000 7fe89cff9700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 0x7fe888032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:24.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.067+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe890040e70 con 0x7fe8a00f51a0 2024-02-19T22:22:24.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.068+0000 7fe89f7fe700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 0x7fe888032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:24.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.068+0000 7fe89f7fe700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 0x7fe888032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe8940096a0 tx=0x7fe894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:24.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.072+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe8900095f0 con 0x7fe8a00f51a0 2024-02-19T22:22:24.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.280+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8a00fad30 con 0x7fe8a00f51a0 2024-02-19T22:22:24.283 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.281+0000 7fe89cff9700 1 -- 172.21.15.203:0/868807466 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe89001cb10 con 0x7fe8a00f51a0 2024-02-19T22:22:24.284 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:24.284 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:24.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 msgr2=0x7fe888032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:24.285 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 0x7fe888032470 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fe8940096a0 tx=0x7fe894006b40).stop 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 msgr2=0x7fe8a0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7fe890009830 tx=0x7fe89000b040).stop 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 shutdown_connections 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe88802ffb0 0x7fe888032470 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 --2- 172.21.15.203:0/868807466 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe8a00f51a0 0x7fe8a0109290 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 >> 172.21.15.203:0/868807466 conn(0x7fe8a00f09a0 msgr2=0x7fe8a00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 shutdown_connections 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:24.283+0000 7fe8a566a700 1 -- 172.21.15.203:0/868807466 wait complete. 2024-02-19T22:22:24.286 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:23.241858+0000 mgr.smithi111.ftjzov (mgr.14162) 281 : cluster [DBG] pgmap v267: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:25.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:25 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:24.282440+0000 mon.smithi111 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.203:0/868807466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:25.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:25.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:25.242144+0000 mgr.smithi111.ftjzov (mgr.14162) 282 : cluster [DBG] pgmap v268: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:27.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 -- 172.21.15.203:0/3417122003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 msgr2=0x7f4ed40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:27.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3417122003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed40f90f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f4ec4009fb0 tx=0x7f4ec4009b30).stop 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 -- 172.21.15.203:0/3417122003 shutdown_connections 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3417122003 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed40f90f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 -- 172.21.15.203:0/3417122003 >> 172.21.15.203:0/3417122003 conn(0x7f4ed40f0b70 msgr2=0x7f4ed40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.989+0000 7f4edb57b700 1 -- 172.21.15.203:0/3417122003 shutdown_connections 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4edb57b700 1 -- 172.21.15.203:0/3417122003 wait complete. 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4edb57b700 1 Processor -- start 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4edb57b700 1 -- start start 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4edb57b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4edb57b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4108e80 con 0x7f4ed4060280 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4ed9317700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4ed9317700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59970/0 (socket says 172.21.15.203:59970) 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.990+0000 7f4ed9317700 1 -- 172.21.15.203:0/3318595937 learned_addr learned my addr 172.21.15.203:0/3318595937 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4ed9317700 1 -- 172.21.15.203:0/3318595937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec4005040 con 0x7f4ed4060280 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4ed9317700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4ec40089f0 tx=0x7f4ec400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:27.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec4009320 con 0x7f4ed4060280 2024-02-19T22:22:27.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed4109080 con 0x7f4ed4060280 2024-02-19T22:22:27.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ec4013380 con 0x7f4ed4060280 2024-02-19T22:22:27.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ec4014e20 con 0x7f4ed4060280 2024-02-19T22:22:27.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.991+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed4109510 con 0x7f4ed4060280 2024-02-19T22:22:27.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.992+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ec40095f0 con 0x7f4ed4060280 2024-02-19T22:22:27.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.992+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed4045ad0 con 0x7f4ed4060280 2024-02-19T22:22:27.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.992+0000 7f4ec9ffb700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 0x7f4ec00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:27.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.992+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ec4040e90 con 0x7f4ed4060280 2024-02-19T22:22:27.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.992+0000 7f4ed8b16700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 0x7f4ec00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:27.996 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.993+0000 7f4ed8b16700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 0x7f4ec00323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4ed00096a0 tx=0x7f4ed0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:27.999 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:27.997+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ec4014420 con 0x7f4ed4060280 2024-02-19T22:22:28.205 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.203+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ed410a3c0 con 0x7f4ed4060280 2024-02-19T22:22:28.206 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.203+0000 7f4ec9ffb700 1 -- 172.21.15.203:0/3318595937 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ec401d980 con 0x7f4ed4060280 2024-02-19T22:22:28.206 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:28.206 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:28.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 msgr2=0x7f4ec00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:28.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 0x7f4ec00323d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4ed00096a0 tx=0x7f4ed0006b40).stop 2024-02-19T22:22:28.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 msgr2=0x7f4ed4108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:28.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f4ec40089f0 tx=0x7f4ec400b040).stop 2024-02-19T22:22:28.208 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 shutdown_connections 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4ec002ff10 0x7f4ec00323d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 --2- 172.21.15.203:0/3318595937 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4ed4060280 0x7f4ed4108990 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 >> 172.21.15.203:0/3318595937 conn(0x7f4ed40f0b70 msgr2=0x7f4ed40f1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 shutdown_connections 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:28.205+0000 7f4edb57b700 1 -- 172.21.15.203:0/3318595937 wait complete. 2024-02-19T22:22:28.209 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:27.242443+0000 mgr.smithi111.ftjzov (mgr.14162) 283 : cluster [DBG] pgmap v269: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:28.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:28.204937+0000 mon.smithi111 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.203:0/3318595937' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:29.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:29.875 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:29.242812+0000 mgr.smithi111.ftjzov (mgr.14162) 284 : cluster [DBG] pgmap v270: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:31.772 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 -- 172.21.15.203:0/3258001065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 msgr2=0x7f2db00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 --2- 172.21.15.203:0/3258001065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db00f6930 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f2d98009fb0 tx=0x7f2d98009b30).stop 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 -- 172.21.15.203:0/3258001065 shutdown_connections 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 --2- 172.21.15.203:0/3258001065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db00f6930 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 -- 172.21.15.203:0/3258001065 >> 172.21.15.203:0/3258001065 conn(0x7f2db00f1c70 msgr2=0x7f2db00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 -- 172.21.15.203:0/3258001065 shutdown_connections 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.768+0000 7f2db5568700 1 -- 172.21.15.203:0/3258001065 wait complete. 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2db5568700 1 Processor -- start 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2db5568700 1 -- start start 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2db5568700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2db5568700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0069e20 con 0x7f2db00f6510 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:31.773 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2daeffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59998/0 (socket says 172.21.15.203:59998) 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.769+0000 7f2daeffd700 1 -- 172.21.15.203:0/1927268776 learned_addr learned my addr 172.21.15.203:0/1927268776 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2daeffd700 1 -- 172.21.15.203:0/1927268776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98005040 con 0x7f2db00f6510 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2daeffd700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2d98005010 tx=0x7f2d9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d98009320 con 0x7f2db00f6510 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db0069fc0 con 0x7f2db00f6510 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d98013380 con 0x7f2db00f6510 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d98014e50 con 0x7f2db00f6510 2024-02-19T22:22:31.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.770+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db006a450 con 0x7f2db00f6510 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2d980095f0 con 0x7f2db00f6510 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0059100 con 0x7f2db00f6510 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2da77fe700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 0x7f2d9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2d98040ee0 con 0x7f2db00f6510 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2dae7fc700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 0x7f2d9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:31.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.771+0000 7f2dae7fc700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 0x7f2d9c0323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:31.778 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.776+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2d98014420 con 0x7f2db00f6510 2024-02-19T22:22:31.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.979+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2db00fc0a0 con 0x7f2db00f6510 2024-02-19T22:22:31.982 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.980+0000 7f2da77fe700 1 -- 172.21.15.203:0/1927268776 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2d9801d980 con 0x7f2db00f6510 2024-02-19T22:22:31.982 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:31.982 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:31.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 msgr2=0x7f2d9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:31.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 0x7f2d9c0323d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).stop 2024-02-19T22:22:31.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 msgr2=0x7f2db006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:31.984 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f2d98005010 tx=0x7f2d9800b040).stop 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 shutdown_connections 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f2d9c02ff10 0x7f2d9c0323d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 --2- 172.21.15.203:0/1927268776 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f2db00f6510 0x7f2db006b9a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 >> 172.21.15.203:0/1927268776 conn(0x7f2db00f1c70 msgr2=0x7f2db00f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 shutdown_connections 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:31.981+0000 7f2db5568700 1 -- 172.21.15.203:0/1927268776 wait complete. 2024-02-19T22:22:31.985 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:32.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:31.243120+0000 mgr.smithi111.ftjzov (mgr.14162) 285 : cluster [DBG] pgmap v271: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:31.981194+0000 mon.smithi111 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.203:0/1927268776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:33.775 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:33.776 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:33.243413+0000 mgr.smithi111.ftjzov (mgr.14162) 286 : cluster [DBG] pgmap v272: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:35.709 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 -- 172.21.15.203:0/2574385880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 msgr2=0x7fba680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 --2- 172.21.15.203:0/2574385880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba680f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7fba5c009fb0 tx=0x7fba5c009b30).stop 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 -- 172.21.15.203:0/2574385880 shutdown_connections 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 --2- 172.21.15.203:0/2574385880 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba680f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 -- 172.21.15.203:0/2574385880 >> 172.21.15.203:0/2574385880 conn(0x7fba680f1c70 msgr2=0x7fba680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 -- 172.21.15.203:0/2574385880 shutdown_connections 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.705+0000 7fba6c881700 1 -- 172.21.15.203:0/2574385880 wait complete. 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6c881700 1 Processor -- start 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6c881700 1 -- start start 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6c881700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6c881700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68106cd0 con 0x7fba680f6510 2024-02-19T22:22:35.710 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:56958/0 (socket says 172.21.15.203:56958) 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6659c700 1 -- 172.21.15.203:0/2195816842 learned_addr learned my addr 172.21.15.203:0/2195816842 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6659c700 1 -- 172.21.15.203:0/2195816842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c005040 con 0x7fba680f6510 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.706+0000 7fba6659c700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fba5c005010 tx=0x7fba5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba5c009320 con 0x7fba680f6510 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba5c013380 con 0x7fba680f6510 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68106ed0 con 0x7fba680f6510 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba5c014e50 con 0x7fba680f6510 2024-02-19T22:22:35.711 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba68107360 con 0x7fba680f6510 2024-02-19T22:22:35.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fba5c0095f0 con 0x7fba680f6510 2024-02-19T22:22:35.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.707+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba68045ad0 con 0x7fba680f6510 2024-02-19T22:22:35.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.708+0000 7fba56ffd700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 0x7fba500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:35.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.708+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fba5c040e90 con 0x7fba680f6510 2024-02-19T22:22:35.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.708+0000 7fba65d9b700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 0x7fba500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:35.713 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.708+0000 7fba65d9b700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 0x7fba500323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fba580096a0 tx=0x7fba58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:35.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.712+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fba5c014420 con 0x7fba680f6510 2024-02-19T22:22:35.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.918+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba68108210 con 0x7fba680f6510 2024-02-19T22:22:35.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.918+0000 7fba56ffd700 1 -- 172.21.15.203:0/2195816842 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fba5c010230 con 0x7fba680f6510 2024-02-19T22:22:35.921 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:35.921 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 msgr2=0x7fba500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 0x7fba500323d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fba580096a0 tx=0x7fba58006b40).stop 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 msgr2=0x7fba681067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7fba5c005010 tx=0x7fba5c00b040).stop 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 shutdown_connections 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fba5002ff10 0x7fba500323d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 --2- 172.21.15.203:0/2195816842 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fba680f6510 0x7fba681067e0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 >> 172.21.15.203:0/2195816842 conn(0x7fba680f1c70 msgr2=0x7fba680f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:35.923 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 shutdown_connections 2024-02-19T22:22:35.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:35.920+0000 7fba6c881700 1 -- 172.21.15.203:0/2195816842 wait complete. 2024-02-19T22:22:35.924 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:36.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:35.243746+0000 mgr.smithi111.ftjzov (mgr.14162) 287 : cluster [DBG] pgmap v273: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:35.919731+0000 mon.smithi111 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.203:0/2195816842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:37.636 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:37.636 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:37.244083+0000 mgr.smithi111.ftjzov (mgr.14162) 288 : cluster [DBG] pgmap v274: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:39.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.544+0000 7f3908913700 1 -- 172.21.15.203:0/3511237365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 msgr2=0x7f39040f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:39.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.544+0000 7f3908913700 1 --2- 172.21.15.203:0/3511237365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39040f90f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f38f8009fb0 tx=0x7f38f8009b30).stop 2024-02-19T22:22:39.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 -- 172.21.15.203:0/3511237365 shutdown_connections 2024-02-19T22:22:39.548 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 --2- 172.21.15.203:0/3511237365 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39040f90f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 -- 172.21.15.203:0/3511237365 >> 172.21.15.203:0/3511237365 conn(0x7f39040f0b70 msgr2=0x7f39040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 -- 172.21.15.203:0/3511237365 shutdown_connections 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 -- 172.21.15.203:0/3511237365 wait complete. 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 Processor -- start 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.545+0000 7f3908913700 1 -- start start 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f3908913700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f3908913700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3904106c90 con 0x7f3904060280 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f390259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f390259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:56972/0 (socket says 172.21.15.203:56972) 2024-02-19T22:22:39.549 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f390259c700 1 -- 172.21.15.203:0/991371228 learned_addr learned my addr 172.21.15.203:0/991371228 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f390259c700 1 -- 172.21.15.203:0/991371228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f8005040 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f390259c700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f38f80089f0 tx=0x7f38f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f8009320 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904106e90 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38f8013380 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38f8014e20 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.546+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3904107320 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.547+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38f80095f0 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.547+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3904045ad0 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.547+0000 7f38f2ffd700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 0x7f38ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.547+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38f8040e90 con 0x7f3904060280 2024-02-19T22:22:39.551 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.547+0000 7f3901d9b700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 0x7f38ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:39.552 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.548+0000 7f3901d9b700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 0x7f38ec0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f38f40096a0 tx=0x7f38f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:39.554 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.552+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38f8014420 con 0x7f3904060280 2024-02-19T22:22:39.757 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.755+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39041081d0 con 0x7f3904060280 2024-02-19T22:22:39.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.755+0000 7f38f2ffd700 1 -- 172.21.15.203:0/991371228 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f38f801d980 con 0x7f3904060280 2024-02-19T22:22:39.758 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:39.758 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:39.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 msgr2=0x7f38ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:39.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 0x7f38ec0323d0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f38f40096a0 tx=0x7f38f4006b40).stop 2024-02-19T22:22:39.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 msgr2=0x7f39041067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:39.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f38f80089f0 tx=0x7f38f800b040).stop 2024-02-19T22:22:39.760 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 shutdown_connections 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f38ec02ff10 0x7f38ec0323d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 --2- 172.21.15.203:0/991371228 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3904060280 0x7f39041067a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 >> 172.21.15.203:0/991371228 conn(0x7f39040f0b70 msgr2=0x7f39040f1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 shutdown_connections 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:39.757+0000 7f3908913700 1 -- 172.21.15.203:0/991371228 wait complete. 2024-02-19T22:22:39.761 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:39.244485+0000 mgr.smithi111.ftjzov (mgr.14162) 289 : cluster [DBG] pgmap v275: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:40.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:39.756795+0000 mon.smithi111 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.203:0/991371228' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:41.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:41.494 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:42.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:41.244759+0000 mgr.smithi111.ftjzov (mgr.14162) 290 : cluster [DBG] pgmap v276: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 -- 172.21.15.203:0/475698317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 msgr2=0x7f01f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 --2- 172.21.15.203:0/475698317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f90f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f01e8009fb0 tx=0x7f01e8009b30).stop 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 -- 172.21.15.203:0/475698317 shutdown_connections 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 --2- 172.21.15.203:0/475698317 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f90f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 -- 172.21.15.203:0/475698317 >> 172.21.15.203:0/475698317 conn(0x7f01f80f0b70 msgr2=0x7f01f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:43.450 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 -- 172.21.15.203:0/475698317 shutdown_connections 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 -- 172.21.15.203:0/475698317 wait complete. 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.446+0000 7f020027d700 1 Processor -- start 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f020027d700 1 -- start start 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f020027d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f020027d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f80f4ff0 con 0x7f01f8060280 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01fe019700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01fe019700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:56984/0 (socket says 172.21.15.203:56984) 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01fe019700 1 -- 172.21.15.203:0/3330292136 learned_addr learned my addr 172.21.15.203:0/3330292136 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01fe019700 1 -- 172.21.15.203:0/3330292136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e8005040 con 0x7f01f8060280 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01fe019700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f01e80089f0 tx=0x7f01e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e8009320 con 0x7f01f8060280 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f80f5190 con 0x7f01f8060280 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.447+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f80f5620 con 0x7f01f8060280 2024-02-19T22:22:43.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.448+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e8013380 con 0x7f01f8060280 2024-02-19T22:22:43.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.448+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e8014d00 con 0x7f01f8060280 2024-02-19T22:22:43.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.448+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e0004fa0 con 0x7f01f8060280 2024-02-19T22:22:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.448+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01e8014420 con 0x7f01f8060280 2024-02-19T22:22:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.448+0000 7f01eeffd700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 0x7f01e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.449+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f01e8040e40 con 0x7f01f8060280 2024-02-19T22:22:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.449+0000 7f01fd818700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 0x7f01e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:43.453 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.449+0000 7f01fd818700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 0x7f01e4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f01f40096a0 tx=0x7f01f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:43.455 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.453+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01e8004490 con 0x7f01f8060280 2024-02-19T22:22:43.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.658+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e0005670 con 0x7f01f8060280 2024-02-19T22:22:43.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.659+0000 7f01eeffd700 1 -- 172.21.15.203:0/3330292136 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f01e8010230 con 0x7f01f8060280 2024-02-19T22:22:43.661 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:43.662 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 msgr2=0x7f01e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 0x7f01e4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f01f40096a0 tx=0x7f01f4006b40).stop 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 msgr2=0x7f01f80f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f01e80089f0 tx=0x7f01e800b040).stop 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 shutdown_connections 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f01e402ff60 0x7f01e4032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:43.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 --2- 172.21.15.203:0/3330292136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f01f8060280 0x7f01f80f6b70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:43.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 >> 172.21.15.203:0/3330292136 conn(0x7f01f80f0b70 msgr2=0x7f01f80f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:43.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 shutdown_connections 2024-02-19T22:22:43.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:43.660+0000 7f020027d700 1 -- 172.21.15.203:0/3330292136 wait complete. 2024-02-19T22:22:43.664 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:43.244975+0000 mgr.smithi111.ftjzov (mgr.14162) 291 : cluster [DBG] pgmap v277: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:44.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:43.660156+0000 mon.smithi111 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.203:0/3330292136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:45.407 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:45.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:45.245316+0000 mgr.smithi111.ftjzov (mgr.14162) 292 : cluster [DBG] pgmap v278: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:47.291 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 -- 172.21.15.203:0/321785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 msgr2=0x7f54800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 --2- 172.21.15.203:0/321785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f5470009fb0 tx=0x7f5470009b30).stop 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 -- 172.21.15.203:0/321785580 shutdown_connections 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 --2- 172.21.15.203:0/321785580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 -- 172.21.15.203:0/321785580 >> 172.21.15.203:0/321785580 conn(0x7f54800f0b70 msgr2=0x7f54800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 -- 172.21.15.203:0/321785580 shutdown_connections 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.289+0000 7f5484fa1700 1 -- 172.21.15.203:0/321785580 wait complete. 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f5484fa1700 1 Processor -- start 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f5484fa1700 1 -- start start 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f5484fa1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:47.292 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f5484fa1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54800fb960 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f547e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f547e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46820/0 (socket says 172.21.15.203:46820) 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.290+0000 7f547e59c700 1 -- 172.21.15.203:0/2882792779 learned_addr learned my addr 172.21.15.203:0/2882792779 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f547e59c700 1 -- 172.21.15.203:0/2882792779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5470005040 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f547e59c700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5470005010 tx=0x7f547000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5470009320 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54800fbb00 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5470013380 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5470014e50 con 0x7f54800f5410 2024-02-19T22:22:47.294 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.291+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54800fbf90 con 0x7f54800f5410 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54700095f0 con 0x7f54800f5410 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5480059100 con 0x7f54800f5410 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f5476ffd700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 0x7f546c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5470040e70 con 0x7f54800f5410 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f547dd9b700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 0x7f546c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:47.295 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.292+0000 7f547dd9b700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 0x7f546c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f54680096a0 tx=0x7f5468006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:47.298 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.296+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5470014420 con 0x7f54800f5410 2024-02-19T22:22:47.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.501+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5480045ad0 con 0x7f54800f5410 2024-02-19T22:22:47.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.502+0000 7f5476ffd700 1 -- 172.21.15.203:0/2882792779 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5470010230 con 0x7f54800f5410 2024-02-19T22:22:47.504 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:47.504 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:47.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 msgr2=0x7f546c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:47.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 0x7f546c0323d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f54680096a0 tx=0x7f5468006b40).stop 2024-02-19T22:22:47.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 msgr2=0x7f54800fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5470005010 tx=0x7f547000b040).stop 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 shutdown_connections 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f546c02ff10 0x7f546c0323d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 --2- 172.21.15.203:0/2882792779 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54800f5410 0x7f54800fd4e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 >> 172.21.15.203:0/2882792779 conn(0x7f54800f0b70 msgr2=0x7f548005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 shutdown_connections 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:47.503+0000 7f5484fa1700 1 -- 172.21.15.203:0/2882792779 wait complete. 2024-02-19T22:22:47.506 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:48.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:47.245714+0000 mgr.smithi111.ftjzov (mgr.14162) 293 : cluster [DBG] pgmap v279: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:48.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:47.502117+0000 mon.smithi111 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.203:0/2882792779' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:49.126 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:49.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:50.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:49.245992+0000 mgr.smithi111.ftjzov (mgr.14162) 294 : cluster [DBG] pgmap v280: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:51.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 -- 172.21.15.203:0/4009323136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 msgr2=0x7f38000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:51.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 --2- 172.21.15.203:0/4009323136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f38000f5830 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f37f4009fb0 tx=0x7f37f4009b30).stop 2024-02-19T22:22:51.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 -- 172.21.15.203:0/4009323136 shutdown_connections 2024-02-19T22:22:51.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 --2- 172.21.15.203:0/4009323136 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f38000f5830 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:51.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 -- 172.21.15.203:0/4009323136 >> 172.21.15.203:0/4009323136 conn(0x7f38000f0b70 msgr2=0x7f38000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:51.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 -- 172.21.15.203:0/4009323136 shutdown_connections 2024-02-19T22:22:51.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 -- 172.21.15.203:0/4009323136 wait complete. 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.136+0000 7f3805926700 1 Processor -- start 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f3805926700 1 -- start start 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f3805926700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f3805926700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3800060240 con 0x7f38000f5410 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37feffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46848/0 (socket says 172.21.15.203:46848) 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37feffd700 1 -- 172.21.15.203:0/2016185938 learned_addr learned my addr 172.21.15.203:0/2016185938 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:51.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37feffd700 1 -- 172.21.15.203:0/2016185938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f4005040 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37feffd700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f37f4005010 tx=0x7f37f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.137+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f4009320 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3800060440 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f37f4013380 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37f4014e50 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3800109bd0 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37f40095f0 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3800045ad0 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.138+0000 7f37e77fe700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 0x7f37e80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.139+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37f4040fa0 con 0x7f38000f5410 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.139+0000 7f37fe7fc700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 0x7f37e80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:51.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.139+0000 7f37fe7fc700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 0x7f37e80323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:51.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.143+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37f4014420 con 0x7f38000f5410 2024-02-19T22:22:51.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.346+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f380010a3f0 con 0x7f38000f5410 2024-02-19T22:22:51.348 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.347+0000 7f37e77fe700 1 -- 172.21.15.203:0/2016185938 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37f4010230 con 0x7f38000f5410 2024-02-19T22:22:51.349 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:51.349 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 msgr2=0x7f37e80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 0x7f37e80323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).stop 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 msgr2=0x7f3800109520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f37f4005010 tx=0x7f37f400b040).stop 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 shutdown_connections 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f37e802ff10 0x7f37e80323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 --2- 172.21.15.203:0/2016185938 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f38000f5410 0x7f3800109520 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 >> 172.21.15.203:0/2016185938 conn(0x7f38000f0b70 msgr2=0x7f38000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 shutdown_connections 2024-02-19T22:22:51.351 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:51.349+0000 7f3805926700 1 -- 172.21.15.203:0/2016185938 wait complete. 2024-02-19T22:22:51.352 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:52.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:51.246259+0000 mgr.smithi111.ftjzov (mgr.14162) 295 : cluster [DBG] pgmap v281: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:52.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:51.347608+0000 mon.smithi111 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.203:0/2016185938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:53.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:53.071 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:53.246509+0000 mgr.smithi111.ftjzov (mgr.14162) 296 : cluster [DBG] pgmap v282: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:54.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 -- 172.21.15.203:0/1712265554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 msgr2=0x7f4e0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:54.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 --2- 172.21.15.203:0/1712265554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c0f5830 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f4e00009fb0 tx=0x7f4e00009b30).stop 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 -- 172.21.15.203:0/1712265554 shutdown_connections 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 --2- 172.21.15.203:0/1712265554 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c0f5830 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 -- 172.21.15.203:0/1712265554 >> 172.21.15.203:0/1712265554 conn(0x7f4e0c0f0b70 msgr2=0x7f4e0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 -- 172.21.15.203:0/1712265554 shutdown_connections 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.918+0000 7f4e11965700 1 -- 172.21.15.203:0/1712265554 wait complete. 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.919+0000 7f4e11965700 1 Processor -- start 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.919+0000 7f4e11965700 1 -- start start 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.919+0000 7f4e11965700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.919+0000 7f4e11965700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c105c10 con 0x7f4e0c0f5410 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e0affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46872/0 (socket says 172.21.15.203:46872) 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e0affd700 1 -- 172.21.15.203:0/2747806767 learned_addr learned my addr 172.21.15.203:0/2747806767 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:54.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e0affd700 1 -- 172.21.15.203:0/2747806767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e00005040 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e0affd700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f4e00005010 tx=0x7f4e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e00009320 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e00013380 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c105e10 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e00014e50 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.920+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c1062a0 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.921+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e000095f0 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.921+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c045ad0 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.921+0000 7f4df37fe700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 0x7f4df40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.921+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e00040e90 con 0x7f4e0c0f5410 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.922+0000 7f4e0a7fc700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 0x7f4df40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.922+0000 7f4e0a7fc700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 0x7f4df40323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4dfc0096a0 tx=0x7f4dfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:54.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:54.926+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e00014420 con 0x7f4e0c0f5410 2024-02-19T22:22:55.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.131+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e0c107150 con 0x7f4e0c0f5410 2024-02-19T22:22:55.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.132+0000 7f4df37fe700 1 -- 172.21.15.203:0/2747806767 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4e00010230 con 0x7f4e0c0f5410 2024-02-19T22:22:55.135 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:55.135 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 msgr2=0x7f4df40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 0x7f4df40323d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f4dfc0096a0 tx=0x7f4dfc006b40).stop 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 msgr2=0x7f4e0c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f4e00005010 tx=0x7f4e0000b040).stop 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 shutdown_connections 2024-02-19T22:22:55.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f4df402ff10 0x7f4df40323d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:55.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 --2- 172.21.15.203:0/2747806767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f4e0c0f5410 0x7f4e0c105720 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:55.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 >> 172.21.15.203:0/2747806767 conn(0x7f4e0c0f0b70 msgr2=0x7f4e0c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:55.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 shutdown_connections 2024-02-19T22:22:55.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:55.134+0000 7f4e11965700 1 -- 172.21.15.203:0/2747806767 wait complete. 2024-02-19T22:22:55.137 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:55.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:55.133306+0000 mon.smithi111 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.203:0/2747806767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:22:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:55.246762+0000 mgr.smithi111.ftjzov (mgr.14162) 297 : cluster [DBG] pgmap v283: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:56.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:22:56.797 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:22:58.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:57.247080+0000 mgr.smithi111.ftjzov (mgr.14162) 298 : cluster [DBG] pgmap v284: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.738+0000 7f04a4e46700 1 -- 172.21.15.203:0/1310306403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 msgr2=0x7f04a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.738+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1310306403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a00f6930 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f0494009fb0 tx=0x7f0494009b30).stop 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 -- 172.21.15.203:0/1310306403 shutdown_connections 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1310306403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a00f6930 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 -- 172.21.15.203:0/1310306403 >> 172.21.15.203:0/1310306403 conn(0x7f04a00f1c70 msgr2=0x7f04a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 -- 172.21.15.203:0/1310306403 shutdown_connections 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 -- 172.21.15.203:0/1310306403 wait complete. 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 Processor -- start 2024-02-19T22:22:58.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.739+0000 7f04a4e46700 1 -- start start 2024-02-19T22:22:58.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f04a4e46700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:58.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f04a4e46700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a0106cb0 con 0x7f04a00f6510 2024-02-19T22:22:58.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f049e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:58.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f049e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54428/0 (socket says 172.21.15.203:54428) 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f049e59c700 1 -- 172.21.15.203:0/1172985827 learned_addr learned my addr 172.21.15.203:0/1172985827 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f049e59c700 1 -- 172.21.15.203:0/1172985827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494005040 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f049e59c700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0494005010 tx=0x7f049400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0494009320 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0494013380 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a0106eb0 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0494014e50 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.740+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a0107340 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.741+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0494014420 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.741+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a0045ad0 con 0x7f04a00f6510 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.741+0000 7f048effd700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 0x7f04880323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:22:58.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.741+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0494040f70 con 0x7f04a00f6510 2024-02-19T22:22:58.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.741+0000 7f049dd9b700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 0x7f04880323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:22:58.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.742+0000 7f049dd9b700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 0x7f04880323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f04900096a0 tx=0x7f0490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:22:58.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.746+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04940095f0 con 0x7f04a00f6510 2024-02-19T22:22:58.953 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.951+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04a01081f0 con 0x7f04a00f6510 2024-02-19T22:22:58.954 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.951+0000 7f048effd700 1 -- 172.21.15.203:0/1172985827 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f049401db70 con 0x7f04a00f6510 2024-02-19T22:22:58.954 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:22:58.954 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:22:58.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 msgr2=0x7f04880323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 0x7f04880323d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f04900096a0 tx=0x7f0490006b40).stop 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 msgr2=0x7f04a01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f0494005010 tx=0x7f049400b040).stop 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 shutdown_connections 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f048802ff10 0x7f04880323d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 --2- 172.21.15.203:0/1172985827 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f04a00f6510 0x7f04a01067c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 >> 172.21.15.203:0/1172985827 conn(0x7f04a00f1c70 msgr2=0x7f04a00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 shutdown_connections 2024-02-19T22:22:58.956 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:22:58.953+0000 7f04a4e46700 1 -- 172.21.15.203:0/1172985827 wait complete. 2024-02-19T22:22:58.957 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:22:59.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:22:59 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:22:58.952681+0000 mon.smithi111 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.203:0/1172985827' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:00.639 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:00.639 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:00.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:22:59.247455+0000 mgr.smithi111.ftjzov (mgr.14162) 299 : cluster [DBG] pgmap v285: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:02.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 -- 172.21.15.203:0/1846747442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 msgr2=0x7f26f40f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:02.616 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/1846747442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f40f4bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f26e4009fb0 tx=0x7f26e4009b30).stop 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 -- 172.21.15.203:0/1846747442 shutdown_connections 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/1846747442 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f40f4bb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 -- 172.21.15.203:0/1846747442 >> 172.21.15.203:0/1846747442 conn(0x7f26f40f0210 msgr2=0x7f26f40f2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 -- 172.21.15.203:0/1846747442 shutdown_connections 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.613+0000 7f26fa7ce700 1 -- 172.21.15.203:0/1846747442 wait complete. 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26fa7ce700 1 Processor -- start 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26fa7ce700 1 -- start start 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26fa7ce700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:02.617 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26fa7ce700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f4106e40 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26f3fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:54440/0 (socket says 172.21.15.203:54440) 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.614+0000 7f26f3fff700 1 -- 172.21.15.203:0/316188683 learned_addr learned my addr 172.21.15.203:0/316188683 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26f3fff700 1 -- 172.21.15.203:0/316188683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e4005040 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26f3fff700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f26e40089f0 tx=0x7f26e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e4009320 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f4107040 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26e4013380 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26e4014e20 con 0x7f26f40f4790 2024-02-19T22:23:02.619 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.615+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41074d0 con 0x7f26f40f4790 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26e40263a0 con 0x7f26f40f4790 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26f0ff9700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 0x7f26dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f26e4040ee0 con 0x7f26f40f4790 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f4045ad0 con 0x7f26f40f4790 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26f37fe700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 0x7f26dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:02.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.616+0000 7f26f37fe700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 0x7f26dc0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f26e80096a0 tx=0x7f26e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:02.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.621+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f26e4012920 con 0x7f26f40f4790 2024-02-19T22:23:02.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:01.247676+0000 mgr.smithi111.ftjzov (mgr.14162) 300 : cluster [DBG] pgmap v286: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:02.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.827+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26f405eb50 con 0x7f26f40f4790 2024-02-19T22:23:02.830 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.828+0000 7f26f0ff9700 1 -- 172.21.15.203:0/316188683 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f26e402b020 con 0x7f26f40f4790 2024-02-19T22:23:02.830 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:02.831 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:02.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 msgr2=0x7f26dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:02.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 0x7f26dc0323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f26e80096a0 tx=0x7f26e8006b40).stop 2024-02-19T22:23:02.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 msgr2=0x7f26f4106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:02.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f26e40089f0 tx=0x7f26e400b040).stop 2024-02-19T22:23:02.832 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 shutdown_connections 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f26dc02ff10 0x7f26dc0323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 --2- 172.21.15.203:0/316188683 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f26f40f4790 0x7f26f4106950 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 >> 172.21.15.203:0/316188683 conn(0x7f26f40f0210 msgr2=0x7f26f40f0c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 shutdown_connections 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:02.829+0000 7f26fa7ce700 1 -- 172.21.15.203:0/316188683 wait complete. 2024-02-19T22:23:02.833 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:03.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:02.829044+0000 mon.smithi111 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.203:0/316188683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:04.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:04.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:04.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:03.248034+0000 mgr.smithi111.ftjzov (mgr.14162) 301 : cluster [DBG] pgmap v287: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 -- 172.21.15.203:0/378010123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 msgr2=0x7fd9740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 --2- 172.21.15.203:0/378010123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9740f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fd964009fb0 tx=0x7fd964009b30).stop 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 -- 172.21.15.203:0/378010123 shutdown_connections 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 --2- 172.21.15.203:0/378010123 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9740f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 -- 172.21.15.203:0/378010123 >> 172.21.15.203:0/378010123 conn(0x7fd9740f1c70 msgr2=0x7fd9740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:06.567 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 -- 172.21.15.203:0/378010123 shutdown_connections 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.563+0000 7fd97a77d700 1 -- 172.21.15.203:0/378010123 wait complete. 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd97a77d700 1 Processor -- start 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd97a77d700 1 -- start start 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd97a77d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd97a77d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd974106cd0 con 0x7fd9740f6510 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd973fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd973fff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60550/0 (socket says 172.21.15.203:60550) 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.564+0000 7fd973fff700 1 -- 172.21.15.203:0/2633557808 learned_addr learned my addr 172.21.15.203:0/2633557808 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd973fff700 1 -- 172.21.15.203:0/2633557808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd964005040 con 0x7fd9740f6510 2024-02-19T22:23:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd973fff700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd964005010 tx=0x7fd96400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd964009320 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd974106ed0 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd964013380 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd964014e50 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.565+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd974107360 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9640095f0 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd974045ad0 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd970ff9700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd964040e90 con 0x7fd9740f6510 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd9737fe700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.566+0000 7fd9737fe700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 0x7fd95c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:06.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.570+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd964014420 con 0x7fd9740f6510 2024-02-19T22:23:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:05.248436+0000 mgr.smithi111.ftjzov (mgr.14162) 302 : cluster [DBG] pgmap v288: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:06.782 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.780+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9741081b0 con 0x7fd9740f6510 2024-02-19T22:23:06.783 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.780+0000 7fd970ff9700 1 -- 172.21.15.203:0/2633557808 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd964010230 con 0x7fd9740f6510 2024-02-19T22:23:06.783 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:06.783 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 msgr2=0x7fd95c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 0x7fd95c0323d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fd9680096a0 tx=0x7fd968006b40).stop 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 msgr2=0x7fd9741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fd964005010 tx=0x7fd96400b040).stop 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 shutdown_connections 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd95c02ff10 0x7fd95c0323d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 --2- 172.21.15.203:0/2633557808 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd9740f6510 0x7fd9741067e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 >> 172.21.15.203:0/2633557808 conn(0x7fd9740f1c70 msgr2=0x7fd9740f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:06.785 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 shutdown_connections 2024-02-19T22:23:06.786 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:06.782+0000 7fd97a77d700 1 -- 172.21.15.203:0/2633557808 wait complete. 2024-02-19T22:23:06.786 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:07.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:07 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:06.781887+0000 mon.smithi111 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.203:0/2633557808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:08.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:08.571 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:07.248789+0000 mgr.smithi111.ftjzov (mgr.14162) 303 : cluster [DBG] pgmap v289: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:10.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 -- 172.21.15.203:0/3458265756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 msgr2=0x7f09000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:10.515 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 --2- 172.21.15.203:0/3458265756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09000f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f08f0009fb0 tx=0x7f08f0009b30).stop 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 -- 172.21.15.203:0/3458265756 shutdown_connections 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 --2- 172.21.15.203:0/3458265756 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09000f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 -- 172.21.15.203:0/3458265756 >> 172.21.15.203:0/3458265756 conn(0x7f09000f0b70 msgr2=0x7f09000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 -- 172.21.15.203:0/3458265756 shutdown_connections 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.511+0000 7f0907f3a700 1 -- 172.21.15.203:0/3458265756 wait complete. 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0907f3a700 1 Processor -- start 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0907f3a700 1 -- start start 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0907f3a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0907f3a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0900106ca0 con 0x7f09000f5410 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0905cd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0905cd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60564/0 (socket says 172.21.15.203:60564) 2024-02-19T22:23:10.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0905cd6700 1 -- 172.21.15.203:0/2578671010 learned_addr learned my addr 172.21.15.203:0/2578671010 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.512+0000 7f0905cd6700 1 -- 172.21.15.203:0/2578671010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f0005040 con 0x7f09000f5410 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f0905cd6700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f08f0005010 tx=0x7f08f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08f0009320 con 0x7f09000f5410 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08f0013380 con 0x7f09000f5410 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0900106ea0 con 0x7f09000f5410 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08f0014e50 con 0x7f09000f5410 2024-02-19T22:23:10.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0900107330 con 0x7f09000f5410 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.513+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08f00095f0 con 0x7f09000f5410 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.514+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0900045ad0 con 0x7f09000f5410 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.514+0000 7f08f67fc700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 0x7f08ec0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.514+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08f0040e90 con 0x7f09000f5410 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.514+0000 7f09054d5700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 0x7f08ec0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:10.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.514+0000 7f09054d5700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 0x7f08ec0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f08fc0096a0 tx=0x7f08fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:10.521 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.518+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08f0014420 con 0x7f09000f5410 2024-02-19T22:23:10.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:09.249174+0000 mgr.smithi111.ftjzov (mgr.14162) 304 : cluster [DBG] pgmap v290: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:10.726 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.724+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09000fafa0 con 0x7f09000f5410 2024-02-19T22:23:10.727 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.724+0000 7f08f67fc700 1 -- 172.21.15.203:0/2578671010 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f08f0010230 con 0x7f09000f5410 2024-02-19T22:23:10.727 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:10.727 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 msgr2=0x7f08ec0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 0x7f08ec0323d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f08fc0096a0 tx=0x7f08fc006b40).stop 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 msgr2=0x7f09001067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f08f0005010 tx=0x7f08f000b040).stop 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 shutdown_connections 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f08ec02ff10 0x7f08ec0323d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 --2- 172.21.15.203:0/2578671010 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f09000f5410 0x7f09001067b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:10.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 >> 172.21.15.203:0/2578671010 conn(0x7f09000f0b70 msgr2=0x7f090005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:10.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 shutdown_connections 2024-02-19T22:23:10.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:10.726+0000 7f0907f3a700 1 -- 172.21.15.203:0/2578671010 wait complete. 2024-02-19T22:23:10.730 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:11.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:10.725988+0000 mon.smithi111 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.203:0/2578671010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:12.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:12.473 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:11.249479+0000 mgr.smithi111.ftjzov (mgr.14162) 305 : cluster [DBG] pgmap v291: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:14.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 -- 172.21.15.203:0/1893930141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 msgr2=0x7fc8200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:14.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 --2- 172.21.15.203:0/1893930141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8200f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fc808009fb0 tx=0x7fc808009b30).stop 2024-02-19T22:23:14.406 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 -- 172.21.15.203:0/1893930141 shutdown_connections 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 --2- 172.21.15.203:0/1893930141 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8200f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 -- 172.21.15.203:0/1893930141 >> 172.21.15.203:0/1893930141 conn(0x7fc8200f1c70 msgr2=0x7fc8200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 -- 172.21.15.203:0/1893930141 shutdown_connections 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 -- 172.21.15.203:0/1893930141 wait complete. 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.403+0000 7fc825eb1700 1 Processor -- start 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc825eb1700 1 -- start start 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc825eb1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:14.407 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc825eb1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc820106cd0 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc81f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc81f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:60580/0 (socket says 172.21.15.203:60580) 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc81f7fe700 1 -- 172.21.15.203:0/297155351 learned_addr learned my addr 172.21.15.203:0/297155351 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc81f7fe700 1 -- 172.21.15.203:0/297155351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808005040 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.404+0000 7fc81f7fe700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc808005010 tx=0x7fc80800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc808009320 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc808013380 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc820106ed0 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc808014e50 con 0x7fc8200f6510 2024-02-19T22:23:14.409 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc820107360 con 0x7fc8200f6510 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.405+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8080263a0 con 0x7fc8200f6510 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.406+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc820045ad0 con 0x7fc8200f6510 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.406+0000 7fc817fff700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 0x7fc80c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.406+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc808040f50 con 0x7fc8200f6510 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.406+0000 7fc81effd700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 0x7fc80c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:14.410 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.406+0000 7fc81effd700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 0x7fc80c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc8100096a0 tx=0x7fc810006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:14.412 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.410+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc80801db70 con 0x7fc8200f6510 2024-02-19T22:23:14.620 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.618+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc820108210 con 0x7fc8200f6510 2024-02-19T22:23:14.621 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.618+0000 7fc817fff700 1 -- 172.21.15.203:0/297155351 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc80801db70 con 0x7fc8200f6510 2024-02-19T22:23:14.621 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:14.621 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:14.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.619+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 msgr2=0x7fc80c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:14.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.619+0000 7fc825eb1700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 0x7fc80c0323d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fc8100096a0 tx=0x7fc810006b40).stop 2024-02-19T22:23:14.622 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.619+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 msgr2=0x7fc8201067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.619+0000 7fc825eb1700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fc808005010 tx=0x7fc80800b040).stop 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 shutdown_connections 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc80c02ff10 0x7fc80c0323d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 --2- 172.21.15.203:0/297155351 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc8200f6510 0x7fc8201067e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 >> 172.21.15.203:0/297155351 conn(0x7fc8200f1c70 msgr2=0x7fc8200f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 shutdown_connections 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:14.620+0000 7fc825eb1700 1 -- 172.21.15.203:0/297155351 wait complete. 2024-02-19T22:23:14.623 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:14.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:13.249760+0000 mgr.smithi111.ftjzov (mgr.14162) 306 : cluster [DBG] pgmap v292: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:15.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:15 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:14.619635+0000 mon.smithi111 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.203:0/297155351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:16.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:16.327 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:15.250162+0000 mgr.smithi111.ftjzov (mgr.14162) 307 : cluster [DBG] pgmap v293: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:18.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 -- 172.21.15.203:0/3795342055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 msgr2=0x7f54c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:18.227 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/3795342055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c80f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f54b0009fb0 tx=0x7f54b0009b30).stop 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 -- 172.21.15.203:0/3795342055 shutdown_connections 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/3795342055 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c80f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 -- 172.21.15.203:0/3795342055 >> 172.21.15.203:0/3795342055 conn(0x7f54c80f0b70 msgr2=0x7f54c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 -- 172.21.15.203:0/3795342055 shutdown_connections 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.224+0000 7f54cdbf6700 1 -- 172.21.15.203:0/3795342055 wait complete. 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54cdbf6700 1 Processor -- start 2024-02-19T22:23:18.228 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54cdbf6700 1 -- start start 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54cdbf6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54cdbf6700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8105c10 con 0x7f54c80f5410 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.225+0000 7f54c77fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55904/0 (socket says 172.21.15.203:55904) 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54c77fe700 1 -- 172.21.15.203:0/4240323915 learned_addr learned my addr 172.21.15.203:0/4240323915 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54c77fe700 1 -- 172.21.15.203:0/4240323915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0005040 con 0x7f54c80f5410 2024-02-19T22:23:18.229 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54c77fe700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f54b0005010 tx=0x7f54b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0009320 con 0x7f54c80f5410 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8105e10 con 0x7f54c80f5410 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54b0013380 con 0x7f54c80f5410 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0014e50 con 0x7f54c80f5410 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.226+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c81062a0 con 0x7f54c80f5410 2024-02-19T22:23:18.230 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.227+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f54b00095f0 con 0x7f54c80f5410 2024-02-19T22:23:18.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.227+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c8045ad0 con 0x7f54c80f5410 2024-02-19T22:23:18.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.227+0000 7f54bffff700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 0x7f54b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:18.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.227+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f54b0040e70 con 0x7f54c80f5410 2024-02-19T22:23:18.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.228+0000 7f54c6ffd700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 0x7f54b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:18.231 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.228+0000 7f54c6ffd700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 0x7f54b40323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f54b80096a0 tx=0x7f54b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:18.234 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.232+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54b0014420 con 0x7f54c80f5410 2024-02-19T22:23:18.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.439+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54c81070f0 con 0x7f54c80f5410 2024-02-19T22:23:18.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.440+0000 7f54bffff700 1 -- 172.21.15.203:0/4240323915 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f54b0010230 con 0x7f54c80f5410 2024-02-19T22:23:18.442 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:18.442 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 msgr2=0x7f54b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 0x7f54b40323d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f54b80096a0 tx=0x7f54b8006b40).stop 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 msgr2=0x7f54c8105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f54b0005010 tx=0x7f54b000b040).stop 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 shutdown_connections 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f54b402ff10 0x7f54b40323d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 --2- 172.21.15.203:0/4240323915 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54c80f5410 0x7f54c8105720 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 >> 172.21.15.203:0/4240323915 conn(0x7f54c80f0b70 msgr2=0x7f54c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:18.444 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 shutdown_connections 2024-02-19T22:23:18.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:18.441+0000 7f54cdbf6700 1 -- 172.21.15.203:0/4240323915 wait complete. 2024-02-19T22:23:18.445 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:18.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:17.250458+0000 mgr.smithi111.ftjzov (mgr.14162) 308 : cluster [DBG] pgmap v294: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:17.305344+0000 mon.smithi111 (mon.0) 291 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:23:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:17.306848+0000 mon.smithi111 (mon.0) 292 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:23:19.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:19 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:18.441061+0000 mon.smithi111 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.203:0/4240323915' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:20.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:20.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:20.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:19.250850+0000 mgr.smithi111.ftjzov (mgr.14162) 309 : cluster [DBG] pgmap v295: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:21.459 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:21 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:21.141981+0000 mon.smithi111 (mon.0) 294 : audit [DBG] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 -- 172.21.15.203:0/3628436139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 msgr2=0x7f089c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 --2- 172.21.15.203:0/3628436139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c0f66c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f088c009fb0 tx=0x7f088c009b30).stop 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 -- 172.21.15.203:0/3628436139 shutdown_connections 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 --2- 172.21.15.203:0/3628436139 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c0f66c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 -- 172.21.15.203:0/3628436139 >> 172.21.15.203:0/3628436139 conn(0x7f089c0f1aa0 msgr2=0x7f089c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 -- 172.21.15.203:0/3628436139 shutdown_connections 2024-02-19T22:23:22.172 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.169+0000 7f089bfff700 1 -- 172.21.15.203:0/3628436139 wait complete. 2024-02-19T22:23:22.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089bfff700 1 Processor -- start 2024-02-19T22:23:22.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089bfff700 1 -- start start 2024-02-19T22:23:22.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089bfff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:22.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089bfff700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089c069e20 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089affd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:55930/0 (socket says 172.21.15.203:55930) 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089affd700 1 -- 172.21.15.203:0/1157746092 learned_addr learned my addr 172.21.15.203:0/1157746092 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089affd700 1 -- 172.21.15.203:0/1157746092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088c005040 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.170+0000 7f089affd700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f088c009830 tx=0x7f088c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f088c009320 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f088c013380 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089c069fc0 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f088c014420 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f089c06a450 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f088c014580 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.171+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f089c059100 con 0x7f089c0f62a0 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.172+0000 7f08837fe700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 0x7f0884032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:22.175 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.172+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f088c040ef0 con 0x7f089c0f62a0 2024-02-19T22:23:22.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.172+0000 7f089a7fc700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 0x7f0884032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:22.176 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.172+0000 7f089a7fc700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 0x7f0884032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f08900096a0 tx=0x7f0890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:22.179 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.176+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f088c01c8b0 con 0x7f089c0f62a0 2024-02-19T22:23:22.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.395+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f089c045ad0 con 0x7f089c0f62a0 2024-02-19T22:23:22.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.396+0000 7f08837fe700 1 -- 172.21.15.203:0/1157746092 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f088c02c2d0 con 0x7f089c0f62a0 2024-02-19T22:23:22.398 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:22.399 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:22.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.397+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 msgr2=0x7f0884032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:22.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.397+0000 7f089bfff700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 0x7f0884032470 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f08900096a0 tx=0x7f0890006b40).stop 2024-02-19T22:23:22.400 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.397+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 msgr2=0x7f089c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.397+0000 7f089bfff700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f088c009830 tx=0x7f088c00b040).stop 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 shutdown_connections 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f088402ffb0 0x7f0884032470 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 --2- 172.21.15.203:0/1157746092 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f089c0f62a0 0x7f089c06b9a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 >> 172.21.15.203:0/1157746092 conn(0x7f089c0f1aa0 msgr2=0x7f089c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 shutdown_connections 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:22.398+0000 7f089bfff700 1 -- 172.21.15.203:0/1157746092 wait complete. 2024-02-19T22:23:22.401 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:23:21.141388+0000 mgr.smithi111.ftjzov (mgr.14162) 310 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi111 ... 2024-02-19T22:23:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:23:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:21:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:22:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:22:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:23:22.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('uid/gid not found') 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: uid/gid not found 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: yield (conn, connr) 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: code, '\n'.join(err))) 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon prometheus.smithi111 ... 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/prometheus:v2.18.1 -e NODE_NAME=smithi111 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/prometheus:v2.18.1 -c %u %g /etc/prometheus 2024-02-19T22:23:22.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Trying to pull docker.io/prom/prometheus:v2.18.1... 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:21:28Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:22:06Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr time="2024-02-19T22:22:43Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: stat: stderr Error: initializing source docker://prom/prometheus:v2.18.1: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/prometheus:v2.18.1: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/prometheus:v2.18.1: reading manifest v2.18.1 in docker.io/prom/prometheus: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: Traceback (most recent call last): 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: main() 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: r = ctx.func(ctx) 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: return func(ctx) 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-02-19T22:23:22.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4185, in extract_uid_gid_monitoring 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: uid, gid = extract_uid_gid(ctx, file_path='/etc/prometheus') 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: File "/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: raise RuntimeError('uid/gid not found') 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: RuntimeError: uid/gid not found 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:21.251062+0000 mgr.smithi111.ftjzov (mgr.14162) 311 : cluster [DBG] pgmap v296: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:21.997963+0000 mon.smithi111 (mon.0) 295 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:22.714 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:22.219297+0000 mon.smithi111 (mon.0) 296 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:23.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:22.397164+0000 mon.smithi111 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.203:0/1157746092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:23.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:23 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:23.005392+0000 mon.smithi111 (mon.0) 298 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:23.251410+0000 mgr.smithi111.ftjzov (mgr.14162) 312 : cluster [DBG] pgmap v297: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:25.387 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:25.387 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:25.251682+0000 mgr.smithi111.ftjzov (mgr.14162) 313 : cluster [DBG] pgmap v298: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:26.282180+0000 mon.smithi111 (mon.0) 299 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- 172.21.15.203:0/1562303831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 msgr2=0x7f76380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 --2- 172.21.15.203:0/1562303831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f76380f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f7628009fb0 tx=0x7f7628009b30).stop 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- 172.21.15.203:0/1562303831 shutdown_connections 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 --2- 172.21.15.203:0/1562303831 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f76380f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- 172.21.15.203:0/1562303831 >> 172.21.15.203:0/1562303831 conn(0x7f76380f0b70 msgr2=0x7f76380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- 172.21.15.203:0/1562303831 shutdown_connections 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- 172.21.15.203:0/1562303831 wait complete. 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 Processor -- start 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- start start 2024-02-19T22:23:27.763 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.760+0000 7f763d598700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638107330 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f7636ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f7636ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37332/0 (socket says 172.21.15.203:37332) 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f7636ffd700 1 -- 172.21.15.203:0/23225541 learned_addr learned my addr 172.21.15.203:0/23225541 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f7636ffd700 1 -- 172.21.15.203:0/23225541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628005040 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f7636ffd700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f7628005010 tx=0x7f762800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7628009320 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7638060220 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7628013380 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7628014e50 con 0x7f76380f5410 2024-02-19T22:23:27.764 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.761+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763810be30 con 0x7f76380f5410 2024-02-19T22:23:27.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.762+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76280095f0 con 0x7f76380f5410 2024-02-19T22:23:27.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.762+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7638045ad0 con 0x7f76380f5410 2024-02-19T22:23:27.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.762+0000 7f761f7fe700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 0x7f76200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:27.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.762+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7628040f70 con 0x7f76380f5410 2024-02-19T22:23:27.765 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.763+0000 7f76367fc700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 0x7f76200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:27.766 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.763+0000 7f76367fc700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 0x7f76200323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f762c0096a0 tx=0x7f762c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:27.769 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.767+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7628014420 con 0x7f76380f5410 2024-02-19T22:23:27.973 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.972+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f763810c620 con 0x7f76380f5410 2024-02-19T22:23:27.974 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.972+0000 7f761f7fe700 1 -- 172.21.15.203:0/23225541 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7628010230 con 0x7f76380f5410 2024-02-19T22:23:27.974 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:27.974 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:27.975 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 msgr2=0x7f76200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 0x7f76200323d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f762c0096a0 tx=0x7f762c006b40).stop 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 msgr2=0x7f763810b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f7628005010 tx=0x7f762800b040).stop 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 shutdown_connections 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f762002ff10 0x7f76200323d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 --2- 172.21.15.203:0/23225541 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76380f5410 0x7f763810b940 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 >> 172.21.15.203:0/23225541 conn(0x7f76380f0b70 msgr2=0x7f76380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 shutdown_connections 2024-02-19T22:23:27.976 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:27.974+0000 7f763d598700 1 -- 172.21.15.203:0/23225541 wait complete. 2024-02-19T22:23:27.977 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:28.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:26.723306+0000 mon.smithi111 (mon.0) 300 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:28.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:26.724077+0000 mgr.smithi111.ftjzov (mgr.14162) 314 : cluster [DBG] pgmap v299: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:28.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:26.726343+0000 mon.smithi111 (mon.0) 301 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' 2024-02-19T22:23:28.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:27 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cephadm 2024-02-19T22:23:26.728489+0000 mgr.smithi111.ftjzov (mgr.14162) 315 : cephadm [INF] Deploying daemon alertmanager.smithi111 on smithi111 2024-02-19T22:23:29.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:27.721826+0000 mon.smithi111 (mon.0) 302 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-02-19T22:23:29.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:27.972884+0000 mon.smithi111 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.203:0/23225541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:29.592 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:29.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:30.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:29 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:28.724441+0000 mgr.smithi111.ftjzov (mgr.14162) 316 : cluster [DBG] pgmap v300: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- 172.21.15.203:0/2683917961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 msgr2=0x7fa0900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2683917961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fa080009fb0 tx=0x7fa080009b30).stop 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- 172.21.15.203:0/2683917961 shutdown_connections 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2683917961 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- 172.21.15.203:0/2683917961 >> 172.21.15.203:0/2683917961 conn(0x7fa0900f0b70 msgr2=0x7fa0900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- 172.21.15.203:0/2683917961 shutdown_connections 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- 172.21.15.203:0/2683917961 wait complete. 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 Processor -- start 2024-02-19T22:23:31.529 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- start start 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.525+0000 7fa0979f5700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0900fb960 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa095791700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa095791700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:37352/0 (socket says 172.21.15.203:37352) 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa095791700 1 -- 172.21.15.203:0/2861225172 learned_addr learned my addr 172.21.15.203:0/2861225172 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa095791700 1 -- 172.21.15.203:0/2861225172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa080005040 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa095791700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fa0800089f0 tx=0x7fa08000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa080009320 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0900fbb00 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa080013380 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.526+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa08001ed80 con 0x7fa0900f5410 2024-02-19T22:23:31.530 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.527+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0900fbf90 con 0x7fa0900f5410 2024-02-19T22:23:31.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.527+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa090059100 con 0x7fa0900f5410 2024-02-19T22:23:31.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.527+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa08001e420 con 0x7fa0900f5410 2024-02-19T22:23:31.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.528+0000 7fa085ffb700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:31.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.528+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa080040ff0 con 0x7fa0900f5410 2024-02-19T22:23:31.531 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.528+0000 7fa094f90700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:31.532 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.528+0000 7fa094f90700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 0x7fa07c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa08c0096a0 tx=0x7fa08c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:31.535 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.532+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0800095f0 con 0x7fa0900f5410 2024-02-19T22:23:31.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.740+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa090045ad0 con 0x7fa0900f5410 2024-02-19T22:23:31.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.740+0000 7fa085ffb700 1 -- 172.21.15.203:0/2861225172 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa08001cb10 con 0x7fa0900f5410 2024-02-19T22:23:31.743 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:31.743 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 msgr2=0x7fa07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 0x7fa07c032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fa08c0096a0 tx=0x7fa08c006b40).stop 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 msgr2=0x7fa0900fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fa0800089f0 tx=0x7fa08000b040).stop 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 shutdown_connections 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa07c02ff60 0x7fa07c032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 --2- 172.21.15.203:0/2861225172 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0900f5410 0x7fa0900fd4e0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 >> 172.21.15.203:0/2861225172 conn(0x7fa0900f0b70 msgr2=0x7fa0900f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 shutdown_connections 2024-02-19T22:23:31.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:31.742+0000 7fa0979f5700 1 -- 172.21.15.203:0/2861225172 wait complete. 2024-02-19T22:23:31.746 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:30.724819+0000 mgr.smithi111.ftjzov (mgr.14162) 317 : cluster [DBG] pgmap v301: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:31.741720+0000 mon.smithi111 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.203:0/2861225172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:33.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:33.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:32.725100+0000 mgr.smithi111.ftjzov (mgr.14162) 318 : cluster [DBG] pgmap v302: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:35.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- 172.21.15.203:0/885178189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 msgr2=0x7fbbe00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:35.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/885178189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe00f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7fbbd0009fb0 tx=0x7fbbd0009b30).stop 2024-02-19T22:23:35.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- 172.21.15.203:0/885178189 shutdown_connections 2024-02-19T22:23:35.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/885178189 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe00f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:35.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- 172.21.15.203:0/885178189 >> 172.21.15.203:0/885178189 conn(0x7fbbe00f09a0 msgr2=0x7fbbe00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- 172.21.15.203:0/885178189 shutdown_connections 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- 172.21.15.203:0/885178189 wait complete. 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 Processor -- start 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 -- start start 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.499+0000 7fbbdf59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbdf59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbe0060240 con 0x7fbbe00f51a0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbde59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbde59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44922/0 (socket says 172.21.15.203:44922) 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbde59c700 1 -- 172.21.15.203:0/4029735531 learned_addr learned my addr 172.21.15.203:0/4029735531 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbde59c700 1 -- 172.21.15.203:0/4029735531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd0005040 con 0x7fbbe00f51a0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbde59c700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fbbd00089f0 tx=0x7fbbd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbd0009320 con 0x7fbbe00f51a0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbd0013380 con 0x7fbbe00f51a0 2024-02-19T22:23:35.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbe0060440 con 0x7fbbe00f51a0 2024-02-19T22:23:35.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbd0014e20 con 0x7fbbe00f51a0 2024-02-19T22:23:35.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.500+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbe0109980 con 0x7fbbe00f51a0 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.501+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbd00263a0 con 0x7fbbe00f51a0 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.501+0000 7fbbceffd700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 0x7fbbc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.501+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbd0040ea0 con 0x7fbbe00f51a0 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.501+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbe0045ad0 con 0x7fbbe00f51a0 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.502+0000 7fbbddd9b700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 0x7fbbc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:35.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.502+0000 7fbbddd9b700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 0x7fbbc80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fbbd40096a0 tx=0x7fbbd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:35.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.506+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbd0012920 con 0x7fbbe00f51a0 2024-02-19T22:23:35.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.711+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbe00fad30 con 0x7fbbe00f51a0 2024-02-19T22:23:35.714 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.712+0000 7fbbceffd700 1 -- 172.21.15.203:0/4029735531 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbd002b020 con 0x7fbbe00f51a0 2024-02-19T22:23:35.715 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:35.715 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.713+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 msgr2=0x7fbbc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.713+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 0x7fbbc80323d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fbbd40096a0 tx=0x7fbbd4006b40).stop 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.713+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 msgr2=0x7fbbe01092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.713+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7fbbd00089f0 tx=0x7fbbd000b040).stop 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 shutdown_connections 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fbbc802ff10 0x7fbbc80323d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 --2- 172.21.15.203:0/4029735531 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fbbe00f51a0 0x7fbbe01092d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 >> 172.21.15.203:0/4029735531 conn(0x7fbbe00f09a0 msgr2=0x7fbbe00f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 shutdown_connections 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:35.714+0000 7fbbdf59e700 1 -- 172.21.15.203:0/4029735531 wait complete. 2024-02-19T22:23:35.717 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:34.725436+0000 mgr.smithi111.ftjzov (mgr.14162) 319 : cluster [DBG] pgmap v303: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:35.713530+0000 mon.smithi111 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.203:0/4029735531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:37.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:37.503 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:36.725751+0000 mgr.smithi111.ftjzov (mgr.14162) 320 : cluster [DBG] pgmap v304: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:39.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 -- 172.21.15.203:0/4181084416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82380f57d0 msgr2=0x7f82380f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:39.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 --2- 172.21.15.203:0/4181084416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82380f57d0 0x7f82380f5bf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f822c009fb0 tx=0x7f822c009b30).stop 2024-02-19T22:23:39.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 -- 172.21.15.203:0/4181084416 shutdown_connections 2024-02-19T22:23:39.516 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 --2- 172.21.15.203:0/4181084416 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82380f57d0 0x7f82380f5bf0 secure :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0x7f822c009fb0 tx=0x7f822c009b30).stop 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 -- 172.21.15.203:0/4181084416 >> 172.21.15.203:0/4181084416 conn(0x7f82380f0120 msgr2=0x7f82380f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 -- 172.21.15.203:0/4181084416 shutdown_connections 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 -- 172.21.15.203:0/4181084416 wait complete. 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.513+0000 7f823ddd8700 1 Processor -- start 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823ddd8700 1 -- start start 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823ddd8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:39.517 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823ddd8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8238106fb0 con 0x7f82381066a0 2024-02-19T22:23:39.518 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823cdd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823cdd6700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44930/0 (socket says 172.21.15.203:44930) 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823cdd6700 1 -- 172.21.15.203:0/2071578255 learned_addr learned my addr 172.21.15.203:0/2071578255 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.514+0000 7f823cdd6700 1 -- 172.21.15.203:0/2071578255 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822c005040 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f823cdd6700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f822c011000 tx=0x7f822c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f822c013380 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f822c013790 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82381071b0 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f822c01ee70 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.515+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8238107d00 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.516+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f822c025060 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.516+0000 7f82357fa700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 0x7f8228032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.516+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8238059100 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.516+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f822c040e80 con 0x7f82381066a0 2024-02-19T22:23:39.519 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.516+0000 7f8237fff700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 0x7f8228032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:39.520 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.517+0000 7f8237fff700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 0x7f8228032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f82240096a0 tx=0x7f8224006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:39.523 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.521+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f822c01c750 con 0x7f82381066a0 2024-02-19T22:23:39.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.727+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8238045ad0 con 0x7f82381066a0 2024-02-19T22:23:39.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.728+0000 7f82357fa700 1 -- 172.21.15.203:0/2071578255 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f822c021e90 con 0x7f82381066a0 2024-02-19T22:23:39.730 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:39.730 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:39.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 msgr2=0x7f8228032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:39.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 0x7f8228032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f82240096a0 tx=0x7f8224006b40).stop 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 msgr2=0x7f8238106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f822c011000 tx=0x7f822c009b30).stop 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 shutdown_connections 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f822802ff60 0x7f8228032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 --2- 172.21.15.203:0/2071578255 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82381066a0 0x7f8238106ac0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 >> 172.21.15.203:0/2071578255 conn(0x7f82380f0120 msgr2=0x7f82380f0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 shutdown_connections 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:39.730+0000 7f823ddd8700 1 -- 172.21.15.203:0/2071578255 wait complete. 2024-02-19T22:23:39.733 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:40.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:38.726107+0000 mgr.smithi111.ftjzov (mgr.14162) 321 : cluster [DBG] pgmap v305: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:40.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:39.729255+0000 mon.smithi111 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.203:0/2071578255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:41.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:41.441 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:42.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:40.726420+0000 mgr.smithi111.ftjzov (mgr.14162) 322 : cluster [DBG] pgmap v306: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:43.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 -- 172.21.15.203:0/561998464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 msgr2=0x7f3f440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/561998464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f440f90f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f3f38009fb0 tx=0x7f3f38009b30).stop 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 -- 172.21.15.203:0/561998464 shutdown_connections 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/561998464 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f440f90f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 -- 172.21.15.203:0/561998464 >> 172.21.15.203:0/561998464 conn(0x7f3f440f0b70 msgr2=0x7f3f440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 -- 172.21.15.203:0/561998464 shutdown_connections 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.439+0000 7f3f4ac85700 1 -- 172.21.15.203:0/561998464 wait complete. 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.440+0000 7f3f4ac85700 1 Processor -- start 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.440+0000 7f3f4ac85700 1 -- start start 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.440+0000 7f3f4ac85700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:43.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.440+0000 7f3f4ac85700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f44106c70 con 0x7f3f44060280 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.440+0000 7f3f48a21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f48a21700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:44946/0 (socket says 172.21.15.203:44946) 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f48a21700 1 -- 172.21.15.203:0/4193226264 learned_addr learned my addr 172.21.15.203:0/4193226264 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f48a21700 1 -- 172.21.15.203:0/4193226264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38005040 con 0x7f3f44060280 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f48a21700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3f380089f0 tx=0x7f3f3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f38009320 con 0x7f3f44060280 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f44106e70 con 0x7f3f44060280 2024-02-19T22:23:43.445 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3f38013380 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f38014e20 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.441+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f44107300 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.442+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3f38014420 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.442+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f44045ad0 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.442+0000 7f3f417fa700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 0x7f3f340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.442+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3f38040ff0 con 0x7f3f44060280 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.442+0000 7f3f43fff700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 0x7f3f340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:43.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.443+0000 7f3f43fff700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 0x7f3f340323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3f300096a0 tx=0x7f3f30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:43.449 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.447+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3f380095f0 con 0x7f3f44060280 2024-02-19T22:23:43.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.651+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3f441081b0 con 0x7f3f44060280 2024-02-19T22:23:43.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.652+0000 7f3f417fa700 1 -- 172.21.15.203:0/4193226264 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3f3801db70 con 0x7f3f44060280 2024-02-19T22:23:43.655 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:43.655 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 msgr2=0x7f3f340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 0x7f3f340323d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3f300096a0 tx=0x7f3f30006b40).stop 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 msgr2=0x7f3f44106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f3f380089f0 tx=0x7f3f3800b040).stop 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 shutdown_connections 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3f3402ff10 0x7f3f340323d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 --2- 172.21.15.203:0/4193226264 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3f44060280 0x7f3f44106780 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 >> 172.21.15.203:0/4193226264 conn(0x7f3f440f0b70 msgr2=0x7f3f440f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 shutdown_connections 2024-02-19T22:23:43.657 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:43.654+0000 7f3f4ac85700 1 -- 172.21.15.203:0/4193226264 wait complete. 2024-02-19T22:23:43.658 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:44.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:42.726638+0000 mgr.smithi111.ftjzov (mgr.14162) 323 : cluster [DBG] pgmap v307: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:44.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:43.653724+0000 mon.smithi111 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.203:0/4193226264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:45.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:45.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:46.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:44.726899+0000 mgr.smithi111.ftjzov (mgr.14162) 324 : cluster [DBG] pgmap v308: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:47.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 -- 172.21.15.203:0/846193726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 msgr2=0x7f7e680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:47.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/846193726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e680f5830 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f7e50009fb0 tx=0x7f7e50009b30).stop 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 -- 172.21.15.203:0/846193726 shutdown_connections 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/846193726 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e680f5830 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 -- 172.21.15.203:0/846193726 >> 172.21.15.203:0/846193726 conn(0x7f7e680f0b70 msgr2=0x7f7e680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 -- 172.21.15.203:0/846193726 shutdown_connections 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.254+0000 7f7e6cc92700 1 -- 172.21.15.203:0/846193726 wait complete. 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6cc92700 1 Processor -- start 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6cc92700 1 -- start start 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6cc92700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:47.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6cc92700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68106cc0 con 0x7f7e680f5410 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41244/0 (socket says 172.21.15.203:41244) 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.255+0000 7f7e6659c700 1 -- 172.21.15.203:0/1556971849 learned_addr learned my addr 172.21.15.203:0/1556971849 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e6659c700 1 -- 172.21.15.203:0/1556971849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e50005040 con 0x7f7e680f5410 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e6659c700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f7e500089f0 tx=0x7f7e5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e50009320 con 0x7f7e680f5410 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e68106ec0 con 0x7f7e680f5410 2024-02-19T22:23:47.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e50013380 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e5001ed80 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.256+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e68107350 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.257+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e5001e420 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.257+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e68045ad0 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.257+0000 7f7e5effd700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 0x7f7e54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.257+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e50041090 con 0x7f7e680f5410 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.257+0000 7f7e65d9b700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 0x7f7e54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:47.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.258+0000 7f7e65d9b700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 0x7f7e54032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7e580096a0 tx=0x7f7e58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:47.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.262+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e500095f0 con 0x7f7e680f5410 2024-02-19T22:23:47.468 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.465+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e68108200 con 0x7f7e680f5410 2024-02-19T22:23:47.469 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.466+0000 7f7e5effd700 1 -- 172.21.15.203:0/1556971849 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e50004490 con 0x7f7e680f5410 2024-02-19T22:23:47.469 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:47.469 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:47.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 msgr2=0x7f7e54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:47.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 0x7f7e54032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f7e580096a0 tx=0x7f7e58006b40).stop 2024-02-19T22:23:47.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 msgr2=0x7f7e681067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:47.470 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f7e500089f0 tx=0x7f7e5000b040).stop 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 shutdown_connections 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7e5402ff60 0x7f7e54032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 --2- 172.21.15.203:0/1556971849 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7e680f5410 0x7f7e681067d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 >> 172.21.15.203:0/1556971849 conn(0x7f7e680f0b70 msgr2=0x7f7e6805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 shutdown_connections 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:47.468+0000 7f7e6cc92700 1 -- 172.21.15.203:0/1556971849 wait complete. 2024-02-19T22:23:47.471 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:48.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:46.727252+0000 mgr.smithi111.ftjzov (mgr.14162) 325 : cluster [DBG] pgmap v309: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:48.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:47.467658+0000 mon.smithi111 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.203:0/1556971849' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:49.083 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:49.083 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:48.727610+0000 mgr.smithi111.ftjzov (mgr.14162) 326 : cluster [DBG] pgmap v310: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 -- 172.21.15.203:0/4230850643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 msgr2=0x7fae340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 --2- 172.21.15.203:0/4230850643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae340f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fae28009fb0 tx=0x7fae28009b30).stop 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 -- 172.21.15.203:0/4230850643 shutdown_connections 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 --2- 172.21.15.203:0/4230850643 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae340f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 -- 172.21.15.203:0/4230850643 >> 172.21.15.203:0/4230850643 conn(0x7fae340f0b70 msgr2=0x7fae340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 -- 172.21.15.203:0/4230850643 shutdown_connections 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.048+0000 7fae38863700 1 -- 172.21.15.203:0/4230850643 wait complete. 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae38863700 1 Processor -- start 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae38863700 1 -- start start 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae38863700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:51.052 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae38863700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae34105bd0 con 0x7fae340f5410 2024-02-19T22:23:51.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae3259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:51.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae3259c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41252/0 (socket says 172.21.15.203:41252) 2024-02-19T22:23:51.053 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae3259c700 1 -- 172.21.15.203:0/2616141733 learned_addr learned my addr 172.21.15.203:0/2616141733 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.049+0000 7fae3259c700 1 -- 172.21.15.203:0/2616141733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28005040 con 0x7fae340f5410 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae3259c700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fae280089f0 tx=0x7fae2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae28009320 con 0x7fae340f5410 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae28013380 con 0x7fae340f5410 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae34105dd0 con 0x7fae340f5410 2024-02-19T22:23:51.054 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae2801ed80 con 0x7fae340f5410 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.050+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae34106260 con 0x7fae340f5410 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fae28014420 con 0x7fae340f5410 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae22ffd700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 0x7fae1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae28040e10 con 0x7fae340f5410 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae34045ad0 con 0x7fae340f5410 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae31d9b700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 0x7fae1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:51.055 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.051+0000 7fae31d9b700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 0x7fae1c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fae240096a0 tx=0x7fae24006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:51.058 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.055+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae280672b0 con 0x7fae340f5410 2024-02-19T22:23:51.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.260+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae34106fb0 con 0x7fae340f5410 2024-02-19T22:23:51.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.261+0000 7fae22ffd700 1 -- 172.21.15.203:0/2616141733 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fae2801c910 con 0x7fae340f5410 2024-02-19T22:23:51.263 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:51.263 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 msgr2=0x7fae1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 0x7fae1c032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fae240096a0 tx=0x7fae24006b40).stop 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 msgr2=0x7fae341056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fae280089f0 tx=0x7fae2800b040).stop 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 shutdown_connections 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fae1c02ff60 0x7fae1c032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 --2- 172.21.15.203:0/2616141733 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fae340f5410 0x7fae341056e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:51.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 >> 172.21.15.203:0/2616141733 conn(0x7fae340f0b70 msgr2=0x7fae340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:51.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 shutdown_connections 2024-02-19T22:23:51.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:51.262+0000 7fae38863700 1 -- 172.21.15.203:0/2616141733 wait complete. 2024-02-19T22:23:51.266 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:52.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:50.727913+0000 mgr.smithi111.ftjzov (mgr.14162) 327 : cluster [DBG] pgmap v311: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:52.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:51.262174+0000 mon.smithi111 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.203:0/2616141733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:52.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:52.960 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:54.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:52.728197+0000 mgr.smithi111.ftjzov (mgr.14162) 328 : cluster [DBG] pgmap v312: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:54.878 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1983388342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 msgr2=0x7f7f8c0f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1983388342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f4c00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f7f74009fb0 tx=0x7f7f74009b30).stop 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1983388342 shutdown_connections 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1983388342 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f4c00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1983388342 >> 172.21.15.203:0/1983388342 conn(0x7f7f8c0f0260 msgr2=0x7f7f8c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1983388342 shutdown_connections 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.875+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1983388342 wait complete. 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f90eb2700 1 Processor -- start 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f90eb2700 1 -- start start 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f90eb2700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f90eb2700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8c0f5850 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f8a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f8a59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:41272/0 (socket says 172.21.15.203:41272) 2024-02-19T22:23:54.879 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f8a59c700 1 -- 172.21.15.203:0/1419810575 learned_addr learned my addr 172.21.15.203:0/1419810575 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.876+0000 7f7f8a59c700 1 -- 172.21.15.203:0/1419810575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f74005040 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f8a59c700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f7f74009830 tx=0x7f7f7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f74009320 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7f74013380 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8c0f59f0 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f74014420 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8c0f5e80 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f7401e420 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.877+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f8c059100 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.878+0000 7f7f82ffd700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.878+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f74040e30 con 0x7f7f8c0f47e0 2024-02-19T22:23:54.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.878+0000 7f7f89d9b700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:54.882 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.878+0000 7f7f89d9b700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 0x7f7f78032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7f7c0096a0 tx=0x7f7f7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:54.885 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:54.882+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f7401cb10 con 0x7f7f8c0f47e0 2024-02-19T22:23:55.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.090+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f8c045ad0 con 0x7f7f8c0f47e0 2024-02-19T22:23:55.093 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.091+0000 7f7f82ffd700 1 -- 172.21.15.203:0/1419810575 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f74004490 con 0x7f7f8c0f47e0 2024-02-19T22:23:55.093 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:55.094 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 msgr2=0x7f7f78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 0x7f7f78032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f7f7c0096a0 tx=0x7f7f7c006b40).stop 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 msgr2=0x7f7f8c0f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f7f74009830 tx=0x7f7f7400b040).stop 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 shutdown_connections 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f7f7802ff60 0x7f7f78032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 --2- 172.21.15.203:0/1419810575 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f7f8c0f47e0 0x7f7f8c0f73d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 >> 172.21.15.203:0/1419810575 conn(0x7f7f8c0f0260 msgr2=0x7f7f8c0f0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 shutdown_connections 2024-02-19T22:23:55.095 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:55.092+0000 7f7f90eb2700 1 -- 172.21.15.203:0/1419810575 wait complete. 2024-02-19T22:23:55.096 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:55.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:55 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:55.092201+0000 mon.smithi111 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.203:0/1419810575' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:23:56.706 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:23:56.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:23:56.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:54.728513+0000 mgr.smithi111.ftjzov (mgr.14162) 329 : cluster [DBG] pgmap v313: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:58.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:56.728897+0000 mgr.smithi111.ftjzov (mgr.14162) 330 : cluster [DBG] pgmap v314: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- 172.21.15.203:0/3556282585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 msgr2=0x7f35340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 --2- 172.21.15.203:0/3556282585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35340f6930 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3524009fb0 tx=0x7f3524009b30).stop 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- 172.21.15.203:0/3556282585 shutdown_connections 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 --2- 172.21.15.203:0/3556282585 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35340f6930 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- 172.21.15.203:0/3556282585 >> 172.21.15.203:0/3556282585 conn(0x7f35340f1c70 msgr2=0x7f35340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- 172.21.15.203:0/3556282585 shutdown_connections 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- 172.21.15.203:0/3556282585 wait complete. 2024-02-19T22:23:58.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 Processor -- start 2024-02-19T22:23:58.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.862+0000 7f353b82b700 1 -- start start 2024-02-19T22:23:58.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f353b82b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f353b82b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534106cd0 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f35395c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f35395c7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:32784/0 (socket says 172.21.15.203:32784) 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f35395c7700 1 -- 172.21.15.203:0/1931570109 learned_addr learned my addr 172.21.15.203:0/1931570109 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f35395c7700 1 -- 172.21.15.203:0/1931570109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524005040 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.863+0000 7f35395c7700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3524005010 tx=0x7f352400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.864+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3524009320 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.864+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3524013380 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.864+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3534106ed0 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.864+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3524014e50 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.864+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3534107360 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.865+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35240263a0 con 0x7f35340f6510 2024-02-19T22:23:58.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.865+0000 7f3529ffb700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:23:58.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.865+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3524040f90 con 0x7f35340f6510 2024-02-19T22:23:58.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.865+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3534045ad0 con 0x7f35340f6510 2024-02-19T22:23:58.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.865+0000 7f3538dc6700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:23:58.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.866+0000 7f3538dc6700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f35300096a0 tx=0x7f3530006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:23:58.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:58.870+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3524012920 con 0x7f35340f6510 2024-02-19T22:23:59.082 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.079+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3534108080 con 0x7f35340f6510 2024-02-19T22:23:59.083 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.080+0000 7f3529ffb700 1 -- 172.21.15.203:0/1931570109 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f352402b020 con 0x7f35340f6510 2024-02-19T22:23:59.083 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:23:59.083 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:23:59.084 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 msgr2=0x7f35200323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f35300096a0 tx=0x7f3530006b40).stop 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 msgr2=0x7f35341067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f3524005010 tx=0x7f352400b040).stop 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 shutdown_connections 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f352002ff10 0x7f35200323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 --2- 172.21.15.203:0/1931570109 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f35340f6510 0x7f35341067e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 >> 172.21.15.203:0/1931570109 conn(0x7f35340f1c70 msgr2=0x7f35340f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 shutdown_connections 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:23:59.082+0000 7f353b82b700 1 -- 172.21.15.203:0/1931570109 wait complete. 2024-02-19T22:23:59.085 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:23:59.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:23:59 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:23:59.081675+0000 mon.smithi111 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.203:0/1931570109' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:00.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:23:58.729167+0000 mgr.smithi111.ftjzov (mgr.14162) 331 : cluster [DBG] pgmap v315: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:00.783 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:00.783 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:02.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:00.729555+0000 mgr.smithi111.ftjzov (mgr.14162) 332 : cluster [DBG] pgmap v316: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 -- 172.21.15.203:0/1721693852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 msgr2=0x7faa000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 --2- 172.21.15.203:0/1721693852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa000f6930 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fa9f4009fb0 tx=0x7fa9f4009b30).stop 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 -- 172.21.15.203:0/1721693852 shutdown_connections 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 --2- 172.21.15.203:0/1721693852 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa000f6930 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 -- 172.21.15.203:0/1721693852 >> 172.21.15.203:0/1721693852 conn(0x7faa000f1c70 msgr2=0x7faa000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 -- 172.21.15.203:0/1721693852 shutdown_connections 2024-02-19T22:24:02.774 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 -- 172.21.15.203:0/1721693852 wait complete. 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.770+0000 7faa06ef7700 1 Processor -- start 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa06ef7700 1 -- start start 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa06ef7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa06ef7700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa00106cd0 con 0x7faa000f6510 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa04c93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa04c93700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:32794/0 (socket says 172.21.15.203:32794) 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.771+0000 7faa04c93700 1 -- 172.21.15.203:0/2637187065 learned_addr learned my addr 172.21.15.203:0/2637187065 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7faa04c93700 1 -- 172.21.15.203:0/2637187065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4005040 con 0x7faa000f6510 2024-02-19T22:24:02.775 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7faa04c93700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa9f4005010 tx=0x7fa9f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:02.776 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4009320 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00106ed0 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9f4013380 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa9f4014e50 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.772+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00107360 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.773+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa9f4014420 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.773+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa00045ad0 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.773+0000 7fa9fd7fa700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 0x7fa9f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.773+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa9f4040e30 con 0x7faa000f6510 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.773+0000 7fa9fffff700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 0x7fa9f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:02.777 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.774+0000 7fa9fffff700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 0x7fa9f00323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:02.780 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.778+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa9f40095f0 con 0x7faa000f6510 2024-02-19T22:24:02.987 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.985+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa00108210 con 0x7faa000f6510 2024-02-19T22:24:02.988 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.985+0000 7fa9fd7fa700 1 -- 172.21.15.203:0/2637187065 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa9f401db70 con 0x7faa000f6510 2024-02-19T22:24:02.988 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:02.988 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 msgr2=0x7fa9f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 0x7fa9f00323d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa9ec0096a0 tx=0x7fa9ec006b40).stop 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 msgr2=0x7faa001067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa9f4005010 tx=0x7fa9f400b040).stop 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 shutdown_connections 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa9f002ff10 0x7fa9f00323d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 --2- 172.21.15.203:0/2637187065 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7faa000f6510 0x7faa001067e0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 >> 172.21.15.203:0/2637187065 conn(0x7faa000f1c70 msgr2=0x7faa000f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 shutdown_connections 2024-02-19T22:24:02.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:02.987+0000 7faa06ef7700 1 -- 172.21.15.203:0/2637187065 wait complete. 2024-02-19T22:24:02.991 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:03.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:03 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:02.986971+0000 mon.smithi111 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.203:0/2637187065' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:04.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:04.619 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:04.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:02.729823+0000 mgr.smithi111.ftjzov (mgr.14162) 333 : cluster [DBG] pgmap v317: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- 172.21.15.203:0/2869681023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 msgr2=0x7fd7840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 --2- 172.21.15.203:0/2869681023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd7840f6930 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fd778009fb0 tx=0x7fd778009b30).stop 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- 172.21.15.203:0/2869681023 shutdown_connections 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 --2- 172.21.15.203:0/2869681023 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd7840f6930 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- 172.21.15.203:0/2869681023 >> 172.21.15.203:0/2869681023 conn(0x7fd7840f1c70 msgr2=0x7fd7840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:06.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- 172.21.15.203:0/2869681023 shutdown_connections 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- 172.21.15.203:0/2869681023 wait complete. 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 Processor -- start 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- start start 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.649+0000 7fd78bdae700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd784069e20 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd789b4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd789b4a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43406/0 (socket says 172.21.15.203:43406) 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd789b4a700 1 -- 172.21.15.203:0/4118212060 learned_addr learned my addr 172.21.15.203:0/4118212060 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd789b4a700 1 -- 172.21.15.203:0/4118212060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd778005040 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd789b4a700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd7780089f0 tx=0x7fd77800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd778009320 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd784069fc0 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd778013380 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd77801ed80 con 0x7fd7840f6510 2024-02-19T22:24:06.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.650+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78406a450 con 0x7fd7840f6510 2024-02-19T22:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.651+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd77801e420 con 0x7fd7840f6510 2024-02-19T22:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.651+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd784059100 con 0x7fd7840f6510 2024-02-19T22:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.651+0000 7fd7767fc700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 0x7fd770032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.651+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd778041010 con 0x7fd7840f6510 2024-02-19T22:24:06.654 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.652+0000 7fd789349700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 0x7fd770032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:06.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.652+0000 7fd789349700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 0x7fd770032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd7800096a0 tx=0x7fd780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:06.658 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.656+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7780095f0 con 0x7fd7840f6510 2024-02-19T22:24:06.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:04.730088+0000 mgr.smithi111.ftjzov (mgr.14162) 334 : cluster [DBG] pgmap v318: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:06.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.863+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7840fc0a0 con 0x7fd7840f6510 2024-02-19T22:24:06.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.864+0000 7fd7767fc700 1 -- 172.21.15.203:0/4118212060 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd77802c2d0 con 0x7fd7840f6510 2024-02-19T22:24:06.866 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:06.866 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 msgr2=0x7fd770032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 0x7fd770032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd7800096a0 tx=0x7fd780006b40).stop 2024-02-19T22:24:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 msgr2=0x7fd78406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd7780089f0 tx=0x7fd77800b040).stop 2024-02-19T22:24:06.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 shutdown_connections 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fd77002ff60 0x7fd770032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 --2- 172.21.15.203:0/4118212060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fd7840f6510 0x7fd78406b9a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 >> 172.21.15.203:0/4118212060 conn(0x7fd7840f1c70 msgr2=0x7fd7840f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 shutdown_connections 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:06.866+0000 7fd78bdae700 1 -- 172.21.15.203:0/4118212060 wait complete. 2024-02-19T22:24:06.868 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:07.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:07 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:06.864452+0000 mon.smithi111 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.203:0/4118212060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:08.553 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:08.553 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:06.730436+0000 mgr.smithi111.ftjzov (mgr.14162) 335 : cluster [DBG] pgmap v319: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:10.439 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 -- 172.21.15.203:0/2657304546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 msgr2=0x7f82700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 --2- 172.21.15.203:0/2657304546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f8260009fb0 tx=0x7f8260009b30).stop 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 -- 172.21.15.203:0/2657304546 shutdown_connections 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 --2- 172.21.15.203:0/2657304546 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 -- 172.21.15.203:0/2657304546 >> 172.21.15.203:0/2657304546 conn(0x7f82700f0b70 msgr2=0x7f82700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 -- 172.21.15.203:0/2657304546 shutdown_connections 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.437+0000 7f8274ec4700 1 -- 172.21.15.203:0/2657304546 wait complete. 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f8274ec4700 1 Processor -- start 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f8274ec4700 1 -- start start 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f8274ec4700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f8274ec4700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82700fb960 con 0x7f82700f5410 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f826e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:10.440 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f826e59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43418/0 (socket says 172.21.15.203:43418) 2024-02-19T22:24:10.441 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f826e59c700 1 -- 172.21.15.203:0/1023227051 learned_addr learned my addr 172.21.15.203:0/1023227051 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.438+0000 7f826e59c700 1 -- 172.21.15.203:0/1023227051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260005040 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f826e59c700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f8260005010 tx=0x7f826000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8260009320 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8260013380 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82700fbb00 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8260014e50 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82700fbf90 con 0x7f82700f5410 2024-02-19T22:24:10.442 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.439+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82600095f0 con 0x7f82700f5410 2024-02-19T22:24:10.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.440+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270059100 con 0x7f82700f5410 2024-02-19T22:24:10.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.440+0000 7f8266ffd700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 0x7f825c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:10.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.440+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8260040e90 con 0x7f82700f5410 2024-02-19T22:24:10.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.440+0000 7f826dd9b700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 0x7f825c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:10.443 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.440+0000 7f826dd9b700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 0x7f825c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f82580096a0 tx=0x7f8258006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:10.446 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.444+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8260014420 con 0x7f82700f5410 2024-02-19T22:24:10.652 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.650+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82700fafa0 con 0x7f82700f5410 2024-02-19T22:24:10.653 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.650+0000 7f8266ffd700 1 -- 172.21.15.203:0/1023227051 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8260010230 con 0x7f82700f5410 2024-02-19T22:24:10.653 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:10.653 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 msgr2=0x7f825c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 0x7f825c0323d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f82580096a0 tx=0x7f8258006b40).stop 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 msgr2=0x7f82700fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f8260005010 tx=0x7f826000b040).stop 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 shutdown_connections 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f825c02ff10 0x7f825c0323d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 --2- 172.21.15.203:0/1023227051 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82700f5410 0x7f82700fd4e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 >> 172.21.15.203:0/1023227051 conn(0x7f82700f0b70 msgr2=0x7f827005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 shutdown_connections 2024-02-19T22:24:10.655 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:10.652+0000 7f8274ec4700 1 -- 172.21.15.203:0/1023227051 wait complete. 2024-02-19T22:24:10.656 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:08.730692+0000 mgr.smithi111.ftjzov (mgr.14162) 336 : cluster [DBG] pgmap v320: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:11.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:11 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:10.652044+0000 mon.smithi111 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.203:0/1023227051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:12.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:12.280 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:12.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:10.731075+0000 mgr.smithi111.ftjzov (mgr.14162) 337 : cluster [DBG] pgmap v321: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:14.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 -- 172.21.15.203:0/1984970330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 msgr2=0x7fb1440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 --2- 172.21.15.203:0/1984970330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1440f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fb138009fb0 tx=0x7fb138009b30).stop 2024-02-19T22:24:14.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 -- 172.21.15.203:0/1984970330 shutdown_connections 2024-02-19T22:24:14.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 --2- 172.21.15.203:0/1984970330 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1440f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 -- 172.21.15.203:0/1984970330 >> 172.21.15.203:0/1984970330 conn(0x7fb1440f0b70 msgr2=0x7fb1440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 -- 172.21.15.203:0/1984970330 shutdown_connections 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.149+0000 7fb149bc0700 1 -- 172.21.15.203:0/1984970330 wait complete. 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.150+0000 7fb149bc0700 1 Processor -- start 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.150+0000 7fb149bc0700 1 -- start start 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.150+0000 7fb149bc0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.150+0000 7fb149bc0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144105bd0 con 0x7fb1440f5410 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb1437fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb1437fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:43444/0 (socket says 172.21.15.203:43444) 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb1437fe700 1 -- 172.21.15.203:0/4228835148 learned_addr learned my addr 172.21.15.203:0/4228835148 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:14.154 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb1437fe700 1 -- 172.21.15.203:0/4228835148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138005040 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb1437fe700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb138005010 tx=0x7fb13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb138009320 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb144105dd0 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb138013380 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb138014e50 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.151+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb144106260 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.152+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1380095f0 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.152+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb144045ad0 con 0x7fb1440f5410 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.152+0000 7fb12bfff700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 0x7fb12c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:14.156 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.152+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb138040e90 con 0x7fb1440f5410 2024-02-19T22:24:14.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.153+0000 7fb142ffd700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 0x7fb12c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:14.157 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.153+0000 7fb142ffd700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 0x7fb12c0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb1340096a0 tx=0x7fb134006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:14.159 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.157+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb138014420 con 0x7fb1440f5410 2024-02-19T22:24:14.363 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.360+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb144107110 con 0x7fb1440f5410 2024-02-19T22:24:14.364 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.361+0000 7fb12bfff700 1 -- 172.21.15.203:0/4228835148 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb138010230 con 0x7fb1440f5410 2024-02-19T22:24:14.364 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:14.364 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 msgr2=0x7fb12c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 0x7fb12c0323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fb1340096a0 tx=0x7fb134006b40).stop 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 msgr2=0x7fb1441056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fb138005010 tx=0x7fb13800b040).stop 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 shutdown_connections 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fb12c02ff10 0x7fb12c0323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 --2- 172.21.15.203:0/4228835148 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fb1440f5410 0x7fb1441056e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 >> 172.21.15.203:0/4228835148 conn(0x7fb1440f0b70 msgr2=0x7fb1440f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:14.366 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 shutdown_connections 2024-02-19T22:24:14.367 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:14.363+0000 7fb149bc0700 1 -- 172.21.15.203:0/4228835148 wait complete. 2024-02-19T22:24:14.367 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:14.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:12.731307+0000 mgr.smithi111.ftjzov (mgr.14162) 338 : cluster [DBG] pgmap v322: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:15.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:15 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:14.362665+0000 mon.smithi111 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.203:0/4228835148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:15.985 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:15.985 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:16.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:14.731585+0000 mgr.smithi111.ftjzov (mgr.14162) 339 : cluster [DBG] pgmap v323: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:17.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.936+0000 7f327a20a700 1 -- 172.21.15.203:0/3805102883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 msgr2=0x7f32740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.936+0000 7f327a20a700 1 --2- 172.21.15.203:0/3805102883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32740f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f3264009fb0 tx=0x7f3264009b30).stop 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.936+0000 7f327a20a700 1 -- 172.21.15.203:0/3805102883 shutdown_connections 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.936+0000 7f327a20a700 1 --2- 172.21.15.203:0/3805102883 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32740f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 -- 172.21.15.203:0/3805102883 >> 172.21.15.203:0/3805102883 conn(0x7f32740f0b70 msgr2=0x7f32740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 -- 172.21.15.203:0/3805102883 shutdown_connections 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 -- 172.21.15.203:0/3805102883 wait complete. 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 Processor -- start 2024-02-19T22:24:17.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 -- start start 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.937+0000 7f327a20a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3274105bd0 con 0x7f32740f5410 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f32737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f32737fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59840/0 (socket says 172.21.15.203:59840) 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f32737fe700 1 -- 172.21.15.203:0/1872699232 learned_addr learned my addr 172.21.15.203:0/1872699232 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f32737fe700 1 -- 172.21.15.203:0/1872699232 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264005040 con 0x7f32740f5410 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f32737fe700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3264005010 tx=0x7f326400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:17.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3264009320 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3264013380 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3274105dd0 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3264014e50 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.938+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3274106260 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.939+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3264014420 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.939+0000 7f325bfff700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 0x7f325c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.939+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3274045ad0 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.939+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3264040eb0 con 0x7f32740f5410 2024-02-19T22:24:17.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.940+0000 7f3272ffd700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 0x7f325c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:17.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.940+0000 7f3272ffd700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 0x7f325c0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f32680096a0 tx=0x7f3268006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:17.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:17.944+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f326401db70 con 0x7f32740f5410 2024-02-19T22:24:18.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.150+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3274107110 con 0x7f32740f5410 2024-02-19T22:24:18.153 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.150+0000 7f325bfff700 1 -- 172.21.15.203:0/1872699232 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3264042030 con 0x7f32740f5410 2024-02-19T22:24:18.153 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:18.153 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 msgr2=0x7f325c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 0x7f325c0323d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f32680096a0 tx=0x7f3268006b40).stop 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 msgr2=0x7f32741056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f3264005010 tx=0x7f326400b040).stop 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 shutdown_connections 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f325c02ff10 0x7f325c0323d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 --2- 172.21.15.203:0/1872699232 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f32740f5410 0x7f32741056e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 >> 172.21.15.203:0/1872699232 conn(0x7f32740f0b70 msgr2=0x7f32740f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 shutdown_connections 2024-02-19T22:24:18.155 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:18.152+0000 7f327a20a700 1 -- 172.21.15.203:0/1872699232 wait complete. 2024-02-19T22:24:18.156 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:16.731917+0000 mgr.smithi111.ftjzov (mgr.14162) 340 : cluster [DBG] pgmap v324: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:17.306580+0000 mon.smithi111 (mon.0) 316 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:24:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:17.307680+0000 mon.smithi111 (mon.0) 317 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:24:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:18.151886+0000 mon.smithi111 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.203:0/1872699232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:19.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:19.893 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:20.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:18.732210+0000 mgr.smithi111.ftjzov (mgr.14162) 341 : cluster [DBG] pgmap v325: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:21.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- 172.21.15.203:0/244426043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 msgr2=0x7f50b00f6b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:21.815 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 --2- 172.21.15.203:0/244426043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b00f6b90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f50a0009fb0 tx=0x7f50a0009b30).stop 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- 172.21.15.203:0/244426043 shutdown_connections 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 --2- 172.21.15.203:0/244426043 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b00f6b90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- 172.21.15.203:0/244426043 >> 172.21.15.203:0/244426043 conn(0x7f50b00f0120 msgr2=0x7f50b00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- 172.21.15.203:0/244426043 shutdown_connections 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- 172.21.15.203:0/244426043 wait complete. 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 Processor -- start 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- start start 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.812+0000 7f50af59e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b0106ad0 con 0x7f50b00f6770 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50ae59c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59868/0 (socket says 172.21.15.203:59868) 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50ae59c700 1 -- 172.21.15.203:0/3044742753 learned_addr learned my addr 172.21.15.203:0/3044742753 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:21.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50ae59c700 1 -- 172.21.15.203:0/3044742753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a0005040 con 0x7f50b00f6770 2024-02-19T22:24:21.817 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50ae59c700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f50a0005010 tx=0x7f50a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50a0009320 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b0106cd0 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50a0013380 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50a0014e50 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.813+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b0107160 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.814+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50a00095f0 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.814+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b0045ad0 con 0x7f50b00f6770 2024-02-19T22:24:21.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.814+0000 7f509effd700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 0x7f50980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:21.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.814+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50a0040e90 con 0x7f50b00f6770 2024-02-19T22:24:21.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.815+0000 7f50add9b700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 0x7f50980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:21.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.815+0000 7f50add9b700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 0x7f50980323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f50a40096a0 tx=0x7f50a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:21.822 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:21.819+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50a0014420 con 0x7f50b00f6770 2024-02-19T22:24:22.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.022+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50b0108010 con 0x7f50b00f6770 2024-02-19T22:24:22.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.023+0000 7f509effd700 1 -- 172.21.15.203:0/3044742753 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50a0010230 con 0x7f50b00f6770 2024-02-19T22:24:22.026 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:22.026 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:22.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.024+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 msgr2=0x7f50980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:22.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.024+0000 7f50af59e700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 0x7f50980323d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f50a40096a0 tx=0x7f50a4006b40).stop 2024-02-19T22:24:22.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.024+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 msgr2=0x7f50b01065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.024+0000 7f50af59e700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f50a0005010 tx=0x7f50a000b040).stop 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 shutdown_connections 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f509802ff10 0x7f50980323d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 --2- 172.21.15.203:0/3044742753 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50b00f6770 0x7f50b01065e0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 >> 172.21.15.203:0/3044742753 conn(0x7f50b00f0120 msgr2=0x7f50b00f4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 shutdown_connections 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:22.025+0000 7f50af59e700 1 -- 172.21.15.203:0/3044742753 wait complete. 2024-02-19T22:24:22.028 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:20.732474+0000 mgr.smithi111.ftjzov (mgr.14162) 342 : cluster [DBG] pgmap v326: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:22.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:22.024462+0000 mon.smithi111 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.203:0/3044742753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:23.751 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:23.751 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:24.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:22.732739+0000 mgr.smithi111.ftjzov (mgr.14162) 343 : cluster [DBG] pgmap v327: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/1149708216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 msgr2=0x7fc1cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/1149708216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fc1b4009fb0 tx=0x7fc1b4009b30).stop 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/1149708216 shutdown_connections 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/1149708216 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/1149708216 >> 172.21.15.203:0/1149708216 conn(0x7fc1cc0f0b70 msgr2=0x7fc1cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/1149708216 shutdown_connections 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.797+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/1149708216 wait complete. 2024-02-19T22:24:25.801 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1d1d9a700 1 Processor -- start 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1d1d9a700 1 -- start start 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1d1d9a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1d1d9a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0f4ff0 con 0x7fc1cc060280 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1cb7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59012/0 (socket says 172.21.15.203:59012) 2024-02-19T22:24:25.802 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.798+0000 7fc1cb7fe700 1 -- 172.21.15.203:0/665538164 learned_addr learned my addr 172.21.15.203:0/665538164 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:25.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1cb7fe700 1 -- 172.21.15.203:0/665538164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b4005040 con 0x7fc1cc060280 2024-02-19T22:24:25.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1cb7fe700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc1b40089f0 tx=0x7fc1b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:25.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b4009320 con 0x7fc1cc060280 2024-02-19T22:24:25.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0f5190 con 0x7fc1cc060280 2024-02-19T22:24:25.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1b4013380 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b4014e20 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.799+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc0f5620 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.800+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1b40095f0 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.800+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc059100 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.800+0000 7fc1c3fff700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 0x7fc1b80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.800+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1b4040f50 con 0x7fc1cc060280 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.800+0000 7fc1caffd700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 0x7fc1b80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:25.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.801+0000 7fc1caffd700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 0x7fc1b80323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:25.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:25.805+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1b4014420 con 0x7fc1cc060280 2024-02-19T22:24:26.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.012+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1cc0fc090 con 0x7fc1cc060280 2024-02-19T22:24:26.015 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.013+0000 7fc1c3fff700 1 -- 172.21.15.203:0/665538164 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc1b401d980 con 0x7fc1cc060280 2024-02-19T22:24:26.015 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:26.016 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 msgr2=0x7fc1b80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 0x7fc1b80323d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fc1bc0096a0 tx=0x7fc1bc006b40).stop 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 msgr2=0x7fc1cc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc1b40089f0 tx=0x7fc1b400b040).stop 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 shutdown_connections 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fc1b802ff10 0x7fc1b80323d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 --2- 172.21.15.203:0/665538164 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fc1cc060280 0x7fc1cc0f6b70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 >> 172.21.15.203:0/665538164 conn(0x7fc1cc0f0b70 msgr2=0x7fc1cc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:26.017 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 shutdown_connections 2024-02-19T22:24:26.018 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:26.014+0000 7fc1d1d9a700 1 -- 172.21.15.203:0/665538164 wait complete. 2024-02-19T22:24:26.018 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:26.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:24.733050+0000 mgr.smithi111.ftjzov (mgr.14162) 344 : cluster [DBG] pgmap v328: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:26.014055+0000 mon.smithi111 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.203:0/665538164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:27.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:27.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:28.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:26.733366+0000 mgr.smithi111.ftjzov (mgr.14162) 345 : cluster [DBG] pgmap v329: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:29.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- 172.21.15.203:0/2506869230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 msgr2=0x7ffbdc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:29.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/2506869230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc0f6930 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7ffbcc009fb0 tx=0x7ffbcc009b30).stop 2024-02-19T22:24:29.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- 172.21.15.203:0/2506869230 shutdown_connections 2024-02-19T22:24:29.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/2506869230 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc0f6930 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:29.689 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- 172.21.15.203:0/2506869230 >> 172.21.15.203:0/2506869230 conn(0x7ffbdc0f1c70 msgr2=0x7ffbdc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- 172.21.15.203:0/2506869230 shutdown_connections 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- 172.21.15.203:0/2506869230 wait complete. 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 Processor -- start 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.685+0000 7ffbe39b0700 1 -- start start 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe39b0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe39b0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc069e20 con 0x7ffbdc0f6510 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe174c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe174c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59036/0 (socket says 172.21.15.203:59036) 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe174c700 1 -- 172.21.15.203:0/3605435290 learned_addr learned my addr 172.21.15.203:0/3605435290 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe174c700 1 -- 172.21.15.203:0/3605435290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc005040 con 0x7ffbdc0f6510 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe174c700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ffbcc005010 tx=0x7ffbcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbcc009320 con 0x7ffbdc0f6510 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffbcc013380 con 0x7ffbdc0f6510 2024-02-19T22:24:29.690 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc069fc0 con 0x7ffbdc0f6510 2024-02-19T22:24:29.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.686+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbcc014e50 con 0x7ffbdc0f6510 2024-02-19T22:24:29.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.687+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc06a450 con 0x7ffbdc0f6510 2024-02-19T22:24:29.691 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.687+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbcc0263a0 con 0x7ffbdc0f6510 2024-02-19T22:24:29.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.688+0000 7ffbd1ffb700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 0x7ffbc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:29.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.688+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbcc040f90 con 0x7ffbdc0f6510 2024-02-19T22:24:29.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.688+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc059100 con 0x7ffbdc0f6510 2024-02-19T22:24:29.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.688+0000 7ffbe0f4b700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 0x7ffbc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:29.692 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.688+0000 7ffbe0f4b700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 0x7ffbc80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ffbd80096a0 tx=0x7ffbd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:29.695 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.693+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbcc012920 con 0x7ffbdc0f6510 2024-02-19T22:24:29.903 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.900+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbdc045ad0 con 0x7ffbdc0f6510 2024-02-19T22:24:29.904 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.902+0000 7ffbd1ffb700 1 -- 172.21.15.203:0/3605435290 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbcc0126f0 con 0x7ffbdc0f6510 2024-02-19T22:24:29.904 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:29.904 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 msgr2=0x7ffbc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 0x7ffbc80323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7ffbd80096a0 tx=0x7ffbd8006b40).stop 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 msgr2=0x7ffbdc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7ffbcc005010 tx=0x7ffbcc00b040).stop 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 shutdown_connections 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ffbc802ff10 0x7ffbc80323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 --2- 172.21.15.203:0/3605435290 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ffbdc0f6510 0x7ffbdc06b9a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 >> 172.21.15.203:0/3605435290 conn(0x7ffbdc0f1c70 msgr2=0x7ffbdc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 shutdown_connections 2024-02-19T22:24:29.906 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:29.903+0000 7ffbe39b0700 1 -- 172.21.15.203:0/3605435290 wait complete. 2024-02-19T22:24:29.907 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:30.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:28.733648+0000 mgr.smithi111.ftjzov (mgr.14162) 346 : cluster [DBG] pgmap v330: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:30.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:29.902445+0000 mon.smithi111 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.203:0/3605435290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:31.673 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:31.674 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:30.733927+0000 mgr.smithi111.ftjzov (mgr.14162) 347 : cluster [DBG] pgmap v331: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:33.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 -- 172.21.15.203:0/440597060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 msgr2=0x7f54080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:33.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 --2- 172.21.15.203:0/440597060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54080f5830 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f53f8009fb0 tx=0x7f53f8009b30).stop 2024-02-19T22:24:33.636 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 -- 172.21.15.203:0/440597060 shutdown_connections 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 --2- 172.21.15.203:0/440597060 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54080f5830 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 -- 172.21.15.203:0/440597060 >> 172.21.15.203:0/440597060 conn(0x7f54080f0b70 msgr2=0x7f54080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 -- 172.21.15.203:0/440597060 shutdown_connections 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.633+0000 7f540cb8b700 1 -- 172.21.15.203:0/440597060 wait complete. 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540cb8b700 1 Processor -- start 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540cb8b700 1 -- start start 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540cb8b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:33.637 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540cb8b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5408105bd0 con 0x7f54080f5410 2024-02-19T22:24:33.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:33.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:59052/0 (socket says 172.21.15.203:59052) 2024-02-19T22:24:33.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.634+0000 7f540659c700 1 -- 172.21.15.203:0/1410070402 learned_addr learned my addr 172.21.15.203:0/1410070402 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:33.638 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f540659c700 1 -- 172.21.15.203:0/1410070402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8005040 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f540659c700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f53f8005010 tx=0x7f53f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f8009320 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5408105dd0 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f53f8013380 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53f8014e50 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.635+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5408106260 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.636+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53f80095f0 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.636+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5408045ad0 con 0x7f54080f5410 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.636+0000 7f53f6ffd700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 0x7f53f00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:33.639 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.636+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53f8040e90 con 0x7f54080f5410 2024-02-19T22:24:33.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.636+0000 7f5405d9b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 0x7f53f00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:33.640 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.637+0000 7f5405d9b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 0x7f53f00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f53fc0096a0 tx=0x7f53fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:33.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.641+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53f8014420 con 0x7f54080f5410 2024-02-19T22:24:33.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.844+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5408107110 con 0x7f54080f5410 2024-02-19T22:24:33.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.845+0000 7f53f6ffd700 1 -- 172.21.15.203:0/1410070402 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53f8010230 con 0x7f54080f5410 2024-02-19T22:24:33.848 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:33.848 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 msgr2=0x7f53f00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 0x7f53f00323d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f53fc0096a0 tx=0x7f53fc006b40).stop 2024-02-19T22:24:33.849 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 msgr2=0x7f54081056e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f53f8005010 tx=0x7f53f800b040).stop 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 shutdown_connections 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f53f002ff10 0x7f53f00323d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 --2- 172.21.15.203:0/1410070402 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f54080f5410 0x7f54081056e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 >> 172.21.15.203:0/1410070402 conn(0x7f54080f0b70 msgr2=0x7f54080f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 shutdown_connections 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:33.847+0000 7f540cb8b700 1 -- 172.21.15.203:0/1410070402 wait complete. 2024-02-19T22:24:33.850 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:32.734184+0000 mgr.smithi111.ftjzov (mgr.14162) 348 : cluster [DBG] pgmap v332: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:33.846536+0000 mon.smithi111 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.203:0/1410070402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:35.562 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:35.562 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:34.734466+0000 mgr.smithi111.ftjzov (mgr.14162) 349 : cluster [DBG] pgmap v333: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.573+0000 7f625d91a700 1 -- 172.21.15.203:0/3154556295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 msgr2=0x7f62580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.573+0000 7f625d91a700 1 --2- 172.21.15.203:0/3154556295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f62580f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f6248009fb0 tx=0x7f6248009b30).stop 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- 172.21.15.203:0/3154556295 shutdown_connections 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 --2- 172.21.15.203:0/3154556295 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f62580f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- 172.21.15.203:0/3154556295 >> 172.21.15.203:0/3154556295 conn(0x7f62580f0b70 msgr2=0x7f62580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- 172.21.15.203:0/3154556295 shutdown_connections 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- 172.21.15.203:0/3154556295 wait complete. 2024-02-19T22:24:37.577 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 Processor -- start 2024-02-19T22:24:37.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- start start 2024-02-19T22:24:37.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:37.578 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.574+0000 7f625d91a700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258106d10 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f6256ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f6256ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49466/0 (socket says 172.21.15.203:49466) 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f6256ffd700 1 -- 172.21.15.203:0/3343222651 learned_addr learned my addr 172.21.15.203:0/3343222651 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f6256ffd700 1 -- 172.21.15.203:0/3343222651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6248005040 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f6256ffd700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6248005010 tx=0x7f624800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6248009320 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6248013380 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258106f10 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6248014e50 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.575+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62581073a0 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.576+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62480263a0 con 0x7f62580f5410 2024-02-19T22:24:37.580 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.576+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6258045ad0 con 0x7f62580f5410 2024-02-19T22:24:37.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.576+0000 7f623f7fe700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:37.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.576+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6248040f00 con 0x7f62580f5410 2024-02-19T22:24:37.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.576+0000 7f62567fc700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:37.581 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.577+0000 7f62567fc700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 0x7f62400323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f624c0096a0 tx=0x7f624c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:37.584 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.581+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f624801dde0 con 0x7f62580f5410 2024-02-19T22:24:37.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.785+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6258108250 con 0x7f62580f5410 2024-02-19T22:24:37.788 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.786+0000 7f623f7fe700 1 -- 172.21.15.203:0/3343222651 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f624801d9d0 con 0x7f62580f5410 2024-02-19T22:24:37.788 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:37.789 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 msgr2=0x7f62400323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 0x7f62400323d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f624c0096a0 tx=0x7f624c006b40).stop 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 msgr2=0x7f6258106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f6248005010 tx=0x7f624800b040).stop 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 shutdown_connections 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f624002ff10 0x7f62400323d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:37.790 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 --2- 172.21.15.203:0/3343222651 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f62580f5410 0x7f6258106820 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:37.791 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 >> 172.21.15.203:0/3343222651 conn(0x7f62580f0b70 msgr2=0x7f62580f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:37.791 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 shutdown_connections 2024-02-19T22:24:37.791 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:37.787+0000 7f625d91a700 1 -- 172.21.15.203:0/3343222651 wait complete. 2024-02-19T22:24:37.791 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:36.734819+0000 mgr.smithi111.ftjzov (mgr.14162) 350 : cluster [DBG] pgmap v334: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:37.787074+0000 mon.smithi111 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.203:0/3343222651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:39.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:39.342 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:40.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:38.735111+0000 mgr.smithi111.ftjzov (mgr.14162) 351 : cluster [DBG] pgmap v335: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 -- 172.21.15.203:0/2473554589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 msgr2=0x7f76440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 --2- 172.21.15.203:0/2473554589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76440f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f762c009fb0 tx=0x7f762c009b30).stop 2024-02-19T22:24:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 -- 172.21.15.203:0/2473554589 shutdown_connections 2024-02-19T22:24:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 --2- 172.21.15.203:0/2473554589 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76440f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:41.167 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 -- 172.21.15.203:0/2473554589 >> 172.21.15.203:0/2473554589 conn(0x7f76440f1c70 msgr2=0x7f76440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:41.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 -- 172.21.15.203:0/2473554589 shutdown_connections 2024-02-19T22:24:41.168 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.164+0000 7f76495eb700 1 -- 172.21.15.203:0/2473554589 wait complete. 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.165+0000 7f76495eb700 1 Processor -- start 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.165+0000 7f76495eb700 1 -- start start 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.165+0000 7f76495eb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.165+0000 7f76495eb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7644106cd0 con 0x7f76440f6510 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.165+0000 7f7642ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f7642ffd700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49488/0 (socket says 172.21.15.203:49488) 2024-02-19T22:24:41.170 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f7642ffd700 1 -- 172.21.15.203:0/1930825619 learned_addr learned my addr 172.21.15.203:0/1930825619 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f7642ffd700 1 -- 172.21.15.203:0/1930825619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762c005040 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f7642ffd700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f762c0089f0 tx=0x7f762c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f762c009320 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7644106ed0 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f762c013380 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f762c01ed80 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.166+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7644107360 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.167+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f762c01e420 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.167+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7644045ad0 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.167+0000 7f763b7fe700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 0x7f7630032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.167+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f762c041120 con 0x7f76440f6510 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.167+0000 7f76427fc700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 0x7f7630032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:41.171 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.168+0000 7f76427fc700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 0x7f7630032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f76340096a0 tx=0x7f7634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:41.174 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.172+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f762c01cb10 con 0x7f76440f6510 2024-02-19T22:24:41.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.380+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7644108210 con 0x7f76440f6510 2024-02-19T22:24:41.383 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.381+0000 7f763b7fe700 1 -- 172.21.15.203:0/1930825619 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f762c02c2d0 con 0x7f76440f6510 2024-02-19T22:24:41.384 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:41.384 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 msgr2=0x7f7630032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 0x7f7630032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f76340096a0 tx=0x7f7634006b40).stop 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 msgr2=0x7f76441067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f762c0089f0 tx=0x7f762c00b040).stop 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 shutdown_connections 2024-02-19T22:24:41.385 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f763002ff60 0x7f7630032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:41.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 --2- 172.21.15.203:0/1930825619 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f76440f6510 0x7f76441067e0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:41.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.382+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 >> 172.21.15.203:0/1930825619 conn(0x7f76440f1c70 msgr2=0x7f76440f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:41.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.383+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 shutdown_connections 2024-02-19T22:24:41.386 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:41.383+0000 7f76495eb700 1 -- 172.21.15.203:0/1930825619 wait complete. 2024-02-19T22:24:41.386 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:42.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:40.735450+0000 mgr.smithi111.ftjzov (mgr.14162) 352 : cluster [DBG] pgmap v336: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:42.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:42 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:41.382326+0000 mon.smithi111 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.203:0/1930825619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:43.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:43.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:44.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:44 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:42.735788+0000 mgr.smithi111.ftjzov (mgr.14162) 353 : cluster [DBG] pgmap v337: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 -- 172.21.15.203:0/1758937593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 msgr2=0x7fca8c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 --2- 172.21.15.203:0/1758937593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c0f66c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fca84009960 tx=0x7fca84004d10).stop 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 -- 172.21.15.203:0/1758937593 shutdown_connections 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 --2- 172.21.15.203:0/1758937593 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c0f66c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 -- 172.21.15.203:0/1758937593 >> 172.21.15.203:0/1758937593 conn(0x7fca8c0f1aa0 msgr2=0x7fca8c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:45.032 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 -- 172.21.15.203:0/1758937593 shutdown_connections 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 -- 172.21.15.203:0/1758937593 wait complete. 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.028+0000 7fca908c0700 1 Processor -- start 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca908c0700 1 -- start start 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca908c0700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca908c0700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8c106ac0 con 0x7fca8c0f62a0 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca8b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:45.033 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca8b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49504/0 (socket says 172.21.15.203:49504) 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca8b7fe700 1 -- 172.21.15.203:0/2461208015 learned_addr learned my addr 172.21.15.203:0/2461208015 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.029+0000 7fca8b7fe700 1 -- 172.21.15.203:0/2461208015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84009640 con 0x7fca8c0f62a0 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca8b7fe700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fca84007bd0 tx=0x7fca84004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca84018070 con 0x7fca8c0f62a0 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca8c106cc0 con 0x7fca8c0f62a0 2024-02-19T22:24:45.034 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fca8400b970 con 0x7fca8c0f62a0 2024-02-19T22:24:45.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca8401d600 con 0x7fca8c0f62a0 2024-02-19T22:24:45.035 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.030+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca8c107150 con 0x7fca8c0f62a0 2024-02-19T22:24:45.036 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fca84024060 con 0x7fca8c0f62a0 2024-02-19T22:24:45.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8c045ad0 con 0x7fca8c0f62a0 2024-02-19T22:24:45.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca83fff700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 0x7fca78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:45.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fca84041110 con 0x7fca8c0f62a0 2024-02-19T22:24:45.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca8affd700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 0x7fca78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:45.037 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.031+0000 7fca8affd700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 0x7fca78032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fca740096a0 tx=0x7fca74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:45.038 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.035+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fca8401bb10 con 0x7fca8c0f62a0 2024-02-19T22:24:45.243 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.242+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca8c107fa0 con 0x7fca8c0f62a0 2024-02-19T22:24:45.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.242+0000 7fca83fff700 1 -- 172.21.15.203:0/2461208015 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fca84027e00 con 0x7fca8c0f62a0 2024-02-19T22:24:45.244 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:45.244 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 msgr2=0x7fca78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 0x7fca78032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fca740096a0 tx=0x7fca74006b40).stop 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 msgr2=0x7fca8c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fca84007bd0 tx=0x7fca84004d10).stop 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 shutdown_connections 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fca7802ff60 0x7fca78032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 --2- 172.21.15.203:0/2461208015 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fca8c0f62a0 0x7fca8c1065d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 >> 172.21.15.203:0/2461208015 conn(0x7fca8c0f1aa0 msgr2=0x7fca8c0f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 shutdown_connections 2024-02-19T22:24:45.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:45.244+0000 7fca908c0700 1 -- 172.21.15.203:0/2461208015 wait complete. 2024-02-19T22:24:45.247 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:45.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:45 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:45.242934+0000 mon.smithi111 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.203:0/2461208015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:46.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:46 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:44.736221+0000 mgr.smithi111.ftjzov (mgr.14162) 354 : cluster [DBG] pgmap v338: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:47.033 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:47.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:48.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:48 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:46.736615+0000 mgr.smithi111.ftjzov (mgr.14162) 355 : cluster [DBG] pgmap v339: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:49.001 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 -- 172.21.15.203:0/2087251663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 msgr2=0x7f85cc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 --2- 172.21.15.203:0/2087251663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc0f6930 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f85bc009fb0 tx=0x7f85bc009b30).stop 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 -- 172.21.15.203:0/2087251663 shutdown_connections 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 --2- 172.21.15.203:0/2087251663 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc0f6930 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 -- 172.21.15.203:0/2087251663 >> 172.21.15.203:0/2087251663 conn(0x7f85cc0f1c70 msgr2=0x7f85cc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 -- 172.21.15.203:0/2087251663 shutdown_connections 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.998+0000 7f85d33ad700 1 -- 172.21.15.203:0/2087251663 wait complete. 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d33ad700 1 Processor -- start 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d33ad700 1 -- start start 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d33ad700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d33ad700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85cc106cd0 con 0x7f85cc0f6510 2024-02-19T22:24:49.002 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d1149700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d1149700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:52800/0 (socket says 172.21.15.203:52800) 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d1149700 1 -- 172.21.15.203:0/3843141004 learned_addr learned my addr 172.21.15.203:0/3843141004 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d1149700 1 -- 172.21.15.203:0/3843141004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85bc005040 con 0x7f85cc0f6510 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:48.999+0000 7f85d1149700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f85bc0089f0 tx=0x7f85bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.000+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85bc009320 con 0x7f85cc0f6510 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.000+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85cc106ed0 con 0x7f85cc0f6510 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.000+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85bc013380 con 0x7f85cc0f6510 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.000+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85bc01ed80 con 0x7f85cc0f6510 2024-02-19T22:24:49.003 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.000+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85cc107360 con 0x7f85cc0f6510 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85bc01e420 con 0x7f85cc0f6510 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85cc045ad0 con 0x7f85cc0f6510 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85c1ffb700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 0x7f85b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85bc040fc0 con 0x7f85cc0f6510 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85d0948700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 0x7f85b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:49.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.001+0000 7f85d0948700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 0x7f85b8032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f85c80096a0 tx=0x7f85c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:49.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.005+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85bc0095f0 con 0x7f85cc0f6510 2024-02-19T22:24:49.211 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.210+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85cc108210 con 0x7f85cc0f6510 2024-02-19T22:24:49.212 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.211+0000 7f85c1ffb700 1 -- 172.21.15.203:0/3843141004 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85bc01cb10 con 0x7f85cc0f6510 2024-02-19T22:24:49.212 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:49.212 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 msgr2=0x7f85b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 0x7f85b8032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f85c80096a0 tx=0x7f85c8006b40).stop 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 msgr2=0x7f85cc1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f85bc0089f0 tx=0x7f85bc00b040).stop 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 shutdown_connections 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f85b802ff60 0x7f85b8032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 --2- 172.21.15.203:0/3843141004 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f85cc0f6510 0x7f85cc1067e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 >> 172.21.15.203:0/3843141004 conn(0x7f85cc0f1c70 msgr2=0x7f85cc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:49.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 shutdown_connections 2024-02-19T22:24:49.215 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:49.212+0000 7f85d33ad700 1 -- 172.21.15.203:0/3843141004 wait complete. 2024-02-19T22:24:49.215 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:49.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:49 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:49.210965+0000 mon.smithi111 (mon.0) 326 : audit [DBG] from='client.? 172.21.15.203:0/3843141004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:50.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:50 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:48.736858+0000 mgr.smithi111.ftjzov (mgr.14162) 356 : cluster [DBG] pgmap v340: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:50.900 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:50.901 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:52.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:52 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:50.737259+0000 mgr.smithi111.ftjzov (mgr.14162) 357 : cluster [DBG] pgmap v341: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:52.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 -- 172.21.15.203:0/3264500552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 msgr2=0x7f3db80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:52.917 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/3264500552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db80f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3dac009fb0 tx=0x7f3dac009b30).stop 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 -- 172.21.15.203:0/3264500552 shutdown_connections 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/3264500552 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db80f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 -- 172.21.15.203:0/3264500552 >> 172.21.15.203:0/3264500552 conn(0x7f3db80f0b70 msgr2=0x7f3db80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 -- 172.21.15.203:0/3264500552 shutdown_connections 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.914+0000 7f3dbed4b700 1 -- 172.21.15.203:0/3264500552 wait complete. 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbed4b700 1 Processor -- start 2024-02-19T22:24:52.918 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbed4b700 1 -- start start 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbed4b700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbed4b700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db8106cd0 con 0x7f3db80f5410 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbcae7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbcae7700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:52810/0 (socket says 172.21.15.203:52810) 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.915+0000 7f3dbcae7700 1 -- 172.21.15.203:0/2774267777 learned_addr learned my addr 172.21.15.203:0/2774267777 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3dbcae7700 1 -- 172.21.15.203:0/2774267777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dac005040 con 0x7f3db80f5410 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3dbcae7700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3dac005010 tx=0x7f3dac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac009320 con 0x7f3db80f5410 2024-02-19T22:24:52.920 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db8106ed0 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dac013380 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3dac014e50 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.916+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db8107360 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3dac0263a0 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3db57fa700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 0x7f3da80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3dac040f50 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3db8045ad0 con 0x7f3db80f5410 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3db7fff700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 0x7f3da80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:52.921 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.917+0000 7f3db7fff700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 0x7f3da80323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3da40096a0 tx=0x7f3da4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:52.924 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:52.922+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3dac012920 con 0x7f3db80f5410 2024-02-19T22:24:53.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.131+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3db80fafa0 con 0x7f3db80f5410 2024-02-19T22:24:53.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.131+0000 7f3db57fa700 1 -- 172.21.15.203:0/2774267777 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3dac02b020 con 0x7f3db80f5410 2024-02-19T22:24:53.134 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:53.134 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 msgr2=0x7f3da80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 0x7f3da80323d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f3da40096a0 tx=0x7f3da4006b40).stop 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 msgr2=0x7f3db81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3dac005010 tx=0x7f3dac00b040).stop 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 shutdown_connections 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f3da802ff10 0x7f3da80323d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 --2- 172.21.15.203:0/2774267777 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f3db80f5410 0x7f3db81067e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 >> 172.21.15.203:0/2774267777 conn(0x7f3db80f0b70 msgr2=0x7f3db80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:53.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 shutdown_connections 2024-02-19T22:24:53.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:53.133+0000 7f3dbed4b700 1 -- 172.21.15.203:0/2774267777 wait complete. 2024-02-19T22:24:53.137 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:53.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:53 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:53.132887+0000 mon.smithi111 (mon.0) 327 : audit [DBG] from='client.? 172.21.15.203:0/2774267777' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:54.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:54 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:52.737547+0000 mgr.smithi111.ftjzov (mgr.14162) 358 : cluster [DBG] pgmap v342: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:54.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:54.811 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:24:56.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:56 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:54.737912+0000 mgr.smithi111.ftjzov (mgr.14162) 359 : cluster [DBG] pgmap v343: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:56.808 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.805+0000 7f506759e700 1 -- 172.21.15.203:0/1877879611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 msgr2=0x7f50680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.805+0000 7f506759e700 1 --2- 172.21.15.203:0/1877879611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f50680f55c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f505c009fb0 tx=0x7f505c009b30).stop 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 -- 172.21.15.203:0/1877879611 shutdown_connections 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 --2- 172.21.15.203:0/1877879611 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f50680f55c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 -- 172.21.15.203:0/1877879611 >> 172.21.15.203:0/1877879611 conn(0x7f50680f09a0 msgr2=0x7f50680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 -- 172.21.15.203:0/1877879611 shutdown_connections 2024-02-19T22:24:56.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 -- 172.21.15.203:0/1877879611 wait complete. 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.806+0000 7f506759e700 1 Processor -- start 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506759e700 1 -- start start 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506759e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506759e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068060280 con 0x7f50680f51a0 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506659c700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33480/0 (socket says 172.21.15.203:33480) 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506659c700 1 -- 172.21.15.203:0/2103474698 learned_addr learned my addr 172.21.15.203:0/2103474698 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:24:56.812 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506659c700 1 -- 172.21.15.203:0/2103474698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c005040 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f506659c700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f505c0089f0 tx=0x7f505c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.807+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c009320 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5068060480 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f505c013380 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f505c014e20 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50681099c0 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f505c0095f0 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.808+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5068045ad0 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.809+0000 7f5056ffd700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.809+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f505c040f50 con 0x7f50680f51a0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.809+0000 7f5065d9b700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:24:56.813 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.809+0000 7f5065d9b700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 0x7f50500323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f50580096a0 tx=0x7f5058006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:24:56.816 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:56.813+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f505c014420 con 0x7f50680f51a0 2024-02-19T22:24:57.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.022+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50680fad30 con 0x7f50680f51a0 2024-02-19T22:24:57.025 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.023+0000 7f5056ffd700 1 -- 172.21.15.203:0/2103474698 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f505c01d980 con 0x7f50680f51a0 2024-02-19T22:24:57.025 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:24:57.026 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 msgr2=0x7f50500323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 0x7f50500323d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f50580096a0 tx=0x7f5058006b40).stop 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 msgr2=0x7f5068109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f505c0089f0 tx=0x7f505c00b040).stop 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 shutdown_connections 2024-02-19T22:24:57.027 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f505002ff10 0x7f50500323d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.024+0000 7f506759e700 1 --2- 172.21.15.203:0/2103474698 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f50680f51a0 0x7f5068109290 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:24:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.025+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 >> 172.21.15.203:0/2103474698 conn(0x7f50680f09a0 msgr2=0x7f50680f13b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:24:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.025+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 shutdown_connections 2024-02-19T22:24:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:24:57.025+0000 7f506759e700 1 -- 172.21.15.203:0/2103474698 wait complete. 2024-02-19T22:24:57.028 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:24:57.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:57 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:24:57.024224+0000 mon.smithi111 (mon.0) 328 : audit [DBG] from='client.? 172.21.15.203:0/2103474698' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:24:58.711 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:24:58 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:56.738226+0000 mgr.smithi111.ftjzov (mgr.14162) 360 : cluster [DBG] pgmap v344: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:24:58.824 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:24:58.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:00.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:00 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:24:58.738499+0000 mgr.smithi111.ftjzov (mgr.14162) 361 : cluster [DBG] pgmap v345: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:00.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 -- 172.21.15.203:0/300757086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 msgr2=0x7f520c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 --2- 172.21.15.203:0/300757086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f5208009fb0 tx=0x7f5208009b30).stop 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 -- 172.21.15.203:0/300757086 shutdown_connections 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 --2- 172.21.15.203:0/300757086 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 -- 172.21.15.203:0/300757086 >> 172.21.15.203:0/300757086 conn(0x7f520c0f0b70 msgr2=0x7f520c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 -- 172.21.15.203:0/300757086 shutdown_connections 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.740+0000 7f5213d63700 1 -- 172.21.15.203:0/300757086 wait complete. 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5213d63700 1 Processor -- start 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5213d63700 1 -- start start 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5213d63700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:00.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5213d63700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520c105bb0 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5211aff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5211aff700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33498/0 (socket says 172.21.15.203:33498) 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5211aff700 1 -- 172.21.15.203:0/2667287403 learned_addr learned my addr 172.21.15.203:0/2667287403 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.741+0000 7f5211aff700 1 -- 172.21.15.203:0/2667287403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5208005040 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f5211aff700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f5208005010 tx=0x7f520800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5208009320 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f520c105db0 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5208013380 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5208014e50 con 0x7f520c0f5410 2024-02-19T22:25:00.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.742+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520c106240 con 0x7f520c0f5410 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f520800f410 con 0x7f520c0f5410 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f52027fc700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 0x7f51f80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5208042f20 con 0x7f520c0f5410 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f520c045ad0 con 0x7f520c0f5410 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f52112fe700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 0x7f51f80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:00.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.743+0000 7f52112fe700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 0x7f51f80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f51fc0096a0 tx=0x7f51fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:00.750 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.748+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5208012920 con 0x7f520c0f5410 2024-02-19T22:25:00.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.955+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f520c106f60 con 0x7f520c0f5410 2024-02-19T22:25:00.958 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.956+0000 7f52027fc700 1 -- 172.21.15.203:0/2667287403 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5208012330 con 0x7f520c0f5410 2024-02-19T22:25:00.958 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:00.959 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 msgr2=0x7f51f80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 0x7f51f80323d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f51fc0096a0 tx=0x7f51fc006b40).stop 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 msgr2=0x7f520c1056c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f5208005010 tx=0x7f520800b040).stop 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 shutdown_connections 2024-02-19T22:25:00.960 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f51f802ff10 0x7f51f80323d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:00.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 --2- 172.21.15.203:0/2667287403 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f520c0f5410 0x7f520c1056c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:00.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 >> 172.21.15.203:0/2667287403 conn(0x7f520c0f0b70 msgr2=0x7f520c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:00.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 shutdown_connections 2024-02-19T22:25:00.961 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:00.957+0000 7f5213d63700 1 -- 172.21.15.203:0/2667287403 wait complete. 2024-02-19T22:25:00.961 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:01.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:01 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:00.957281+0000 mon.smithi111 (mon.0) 329 : audit [DBG] from='client.? 172.21.15.203:0/2667287403' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:02.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:02 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:00.738827+0000 mgr.smithi111.ftjzov (mgr.14162) 362 : cluster [DBG] pgmap v346: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:02.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:02.730 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/1744362044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 msgr2=0x7fa0c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/1744362044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c80f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa0c0009960 tx=0x7fa0c0004d10).stop 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/1744362044 shutdown_connections 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/1744362044 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c80f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/1744362044 >> 172.21.15.203:0/1744362044 conn(0x7fa0c80f0b70 msgr2=0x7fa0c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/1744362044 shutdown_connections 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.691+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/1744362044 wait complete. 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0ceaf1700 1 Processor -- start 2024-02-19T22:25:04.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0ceaf1700 1 -- start start 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0ceaf1700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0ceaf1700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c8060220 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0cc88d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0cc88d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:33514/0 (socket says 172.21.15.203:33514) 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.692+0000 7fa0cc88d700 1 -- 172.21.15.203:0/3276633580 learned_addr learned my addr 172.21.15.203:0/3276633580 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0cc88d700 1 -- 172.21.15.203:0/3276633580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c0009640 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0cc88d700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa0c0004b70 tx=0x7fa0c0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c0018070 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c8060420 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa0c000b970 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0c001c560 con 0x7fa0c80f5410 2024-02-19T22:25:04.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.693+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c8109bb0 con 0x7fa0c80f5410 2024-02-19T22:25:04.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa0c001c6c0 con 0x7fa0c80f5410 2024-02-19T22:25:04.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c8059100 con 0x7fa0c80f5410 2024-02-19T22:25:04.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0c57fa700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 0x7fa0b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:04.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa0c00415a0 con 0x7fa0c80f5410 2024-02-19T22:25:04.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0c7fff700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 0x7fa0b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:04.699 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.694+0000 7fa0c7fff700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 0x7fa0b0032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa0bc0096a0 tx=0x7fa0bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:04.701 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.698+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0c00127f0 con 0x7fa0c80f5410 2024-02-19T22:25:04.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:04 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:02.739204+0000 mgr.smithi111.ftjzov (mgr.14162) 363 : cluster [DBG] pgmap v347: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:04.907 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.904+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa0c80fafa0 con 0x7fa0c80f5410 2024-02-19T22:25:04.908 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.905+0000 7fa0c57fa700 1 -- 172.21.15.203:0/3276633580 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa0c00127f0 con 0x7fa0c80f5410 2024-02-19T22:25:04.908 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:04.908 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:04.909 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.906+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 msgr2=0x7fa0b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.906+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 0x7fa0b0032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa0bc0096a0 tx=0x7fa0bc006b40).stop 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.906+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 msgr2=0x7fa0c8109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.906+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa0c0004b70 tx=0x7fa0c0004d10).stop 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 shutdown_connections 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa0b002ff60 0x7fa0b0032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 --2- 172.21.15.203:0/3276633580 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa0c80f5410 0x7fa0c8109500 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 >> 172.21.15.203:0/3276633580 conn(0x7fa0c80f0b70 msgr2=0x7fa0c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 shutdown_connections 2024-02-19T22:25:04.910 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:04.907+0000 7fa0ceaf1700 1 -- 172.21.15.203:0/3276633580 wait complete. 2024-02-19T22:25:04.911 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:05.672 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:05 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:04.906546+0000 mon.smithi111 (mon.0) 330 : audit [DBG] from='client.? 172.21.15.203:0/3276633580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:06.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:06.635 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:06.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:06 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:04.739591+0000 mgr.smithi111.ftjzov (mgr.14162) 364 : cluster [DBG] pgmap v348: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:08.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- 172.21.15.203:0/1975585428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 msgr2=0x7fe510069e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:08.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 --2- 172.21.15.203:0/1975585428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe510069e40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe500009fb0 tx=0x7fe500009b30).stop 2024-02-19T22:25:08.696 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- 172.21.15.203:0/1975585428 shutdown_connections 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 --2- 172.21.15.203:0/1975585428 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe510069e40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- 172.21.15.203:0/1975585428 >> 172.21.15.203:0/1975585428 conn(0x7fe5100f1ac0 msgr2=0x7fe5100f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- 172.21.15.203:0/1975585428 shutdown_connections 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- 172.21.15.203:0/1975585428 wait complete. 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 Processor -- start 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.669+0000 7fe514c4d700 1 -- start start 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe514c4d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe514c4d700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510108cc0 con 0x7fe51006b8a0 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50f7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49110/0 (socket says 172.21.15.203:49110) 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50f7fe700 1 -- 172.21.15.203:0/2822052694 learned_addr learned my addr 172.21.15.203:0/2822052694 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50f7fe700 1 -- 172.21.15.203:0/2822052694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe500005040 con 0x7fe51006b8a0 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50f7fe700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fe500005010 tx=0x7fe50000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe500009320 con 0x7fe51006b8a0 2024-02-19T22:25:08.697 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe510108ec0 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe500013380 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe500014e50 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.670+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510109350 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.671+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5000095f0 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.671+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe510045ad0 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.671+0000 7fe50cff9700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 0x7fe4fc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.671+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe500040fa0 con 0x7fe51006b8a0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.671+0000 7fe507fff700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 0x7fe4fc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.672+0000 7fe507fff700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 0x7fe4fc0323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe4f80096a0 tx=0x7fe4f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:08.698 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.676+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe500014420 con 0x7fe51006b8a0 2024-02-19T22:25:08.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:08 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:06.739889+0000 mgr.smithi111.ftjzov (mgr.14162) 365 : cluster [DBG] pgmap v349: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:08.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.885+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe51010a200 con 0x7fe51006b8a0 2024-02-19T22:25:08.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.885+0000 7fe50cff9700 1 -- 172.21.15.203:0/2822052694 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe500010230 con 0x7fe51006b8a0 2024-02-19T22:25:08.888 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:08.888 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 msgr2=0x7fe4fc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 0x7fe4fc0323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fe4f80096a0 tx=0x7fe4f8006b40).stop 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 msgr2=0x7fe5101087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fe500005010 tx=0x7fe50000b040).stop 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 shutdown_connections 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fe4fc02ff10 0x7fe4fc0323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 --2- 172.21.15.203:0/2822052694 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fe51006b8a0 0x7fe5101087d0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 >> 172.21.15.203:0/2822052694 conn(0x7fe5100f1ac0 msgr2=0x7fe5100f2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 shutdown_connections 2024-02-19T22:25:08.890 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:08.887+0000 7fe514c4d700 1 -- 172.21.15.203:0/2822052694 wait complete. 2024-02-19T22:25:08.891 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:09.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:09 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:08.886889+0000 mon.smithi111 (mon.0) 331 : audit [DBG] from='client.? 172.21.15.203:0/2822052694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:10.623 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:10.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:10.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:10 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:08.740282+0000 mgr.smithi111.ftjzov (mgr.14162) 366 : cluster [DBG] pgmap v350: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- 172.21.15.203:0/3351210491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 msgr2=0x7f495c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 --2- 172.21.15.203:0/3351210491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c0f55c0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f4958009fb0 tx=0x7f4958009b30).stop 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- 172.21.15.203:0/3351210491 shutdown_connections 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 --2- 172.21.15.203:0/3351210491 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c0f55c0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- 172.21.15.203:0/3351210491 >> 172.21.15.203:0/3351210491 conn(0x7f495c0f09a0 msgr2=0x7f495c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- 172.21.15.203:0/3351210491 shutdown_connections 2024-02-19T22:25:12.542 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- 172.21.15.203:0/3351210491 wait complete. 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 Processor -- start 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- start start 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.538+0000 7f4962edb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f495c106ac0 con 0x7f495c0f51a0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4961ed9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4961ed9700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:49126/0 (socket says 172.21.15.203:49126) 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4961ed9700 1 -- 172.21.15.203:0/3186376024 learned_addr learned my addr 172.21.15.203:0/3186376024 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4961ed9700 1 -- 172.21.15.203:0/3186376024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958005040 con 0x7f495c0f51a0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4961ed9700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f49580089f0 tx=0x7f495800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4958009320 con 0x7f495c0f51a0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f495c106cc0 con 0x7f495c0f51a0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4958013380 con 0x7f495c0f51a0 2024-02-19T22:25:12.543 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4958014e20 con 0x7f495c0f51a0 2024-02-19T22:25:12.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.539+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495c107150 con 0x7f495c0f51a0 2024-02-19T22:25:12.544 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.540+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49580095f0 con 0x7f495c0f51a0 2024-02-19T22:25:12.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.540+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495c045ad0 con 0x7f495c0f51a0 2024-02-19T22:25:12.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.540+0000 7f49527fc700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 0x7f49480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:12.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.540+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4958040e90 con 0x7f495c0f51a0 2024-02-19T22:25:12.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.541+0000 7f49616d8700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 0x7f49480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:12.545 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.541+0000 7f49616d8700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 0x7f49480323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f494c0096a0 tx=0x7f494c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:12.547 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.545+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4958014420 con 0x7f495c0f51a0 2024-02-19T22:25:12.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:12 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:10.740681+0000 mgr.smithi111.ftjzov (mgr.14162) 367 : cluster [DBG] pgmap v351: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:12.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.752+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f495c0fad30 con 0x7f495c0f51a0 2024-02-19T22:25:12.755 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.753+0000 7f49527fc700 1 -- 172.21.15.203:0/3186376024 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f495801d980 con 0x7f495c0f51a0 2024-02-19T22:25:12.756 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:12.756 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:12.757 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 msgr2=0x7f49480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:12.757 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 0x7f49480323d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f494c0096a0 tx=0x7f494c006b40).stop 2024-02-19T22:25:12.757 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 msgr2=0x7f495c1065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:12.757 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f49580089f0 tx=0x7f495800b040).stop 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 shutdown_connections 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f494802ff10 0x7f49480323d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 --2- 172.21.15.203:0/3186376024 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f495c0f51a0 0x7f495c1065d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.754+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 >> 172.21.15.203:0/3186376024 conn(0x7f495c0f09a0 msgr2=0x7f495c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.755+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 shutdown_connections 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:12.755+0000 7f4962edb700 1 -- 172.21.15.203:0/3186376024 wait complete. 2024-02-19T22:25:12.758 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:13.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:13 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:12.754328+0000 mon.smithi111 (mon.0) 332 : audit [DBG] from='client.? 172.21.15.203:0/3186376024' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:14.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:14.517 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:14.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:14 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:12.741086+0000 mgr.smithi111.ftjzov (mgr.14162) 368 : cluster [DBG] pgmap v352: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:16.389 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 -- 172.21.15.203:0/3911755678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 msgr2=0x7fa02c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 --2- 172.21.15.203:0/3911755678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c0f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fa020009fb0 tx=0x7fa020009b30).stop 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 -- 172.21.15.203:0/3911755678 shutdown_connections 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 --2- 172.21.15.203:0/3911755678 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c0f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 -- 172.21.15.203:0/3911755678 >> 172.21.15.203:0/3911755678 conn(0x7fa02c0f0b70 msgr2=0x7fa02c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 -- 172.21.15.203:0/3911755678 shutdown_connections 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.386+0000 7fa031b6e700 1 -- 172.21.15.203:0/3911755678 wait complete. 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa031b6e700 1 Processor -- start 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa031b6e700 1 -- start start 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa031b6e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:16.390 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa031b6e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c060220 con 0x7fa02c0f5410 2024-02-19T22:25:16.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa02b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:16.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa02b7fe700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46092/0 (socket says 172.21.15.203:46092) 2024-02-19T22:25:16.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.387+0000 7fa02b7fe700 1 -- 172.21.15.203:0/240050258 learned_addr learned my addr 172.21.15.203:0/240050258 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:16.392 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa02b7fe700 1 -- 172.21.15.203:0/240050258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020005040 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa02b7fe700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa020005010 tx=0x7fa02000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa020009320 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c060420 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa020013380 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa020014e50 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.388+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c109bb0 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.389+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa020014420 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.389+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c059100 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.389+0000 7fa013fff700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 0x7fa0140323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.389+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa020040eb0 con 0x7fa02c0f5410 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.389+0000 7fa02affd700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 0x7fa0140323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:16.393 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.390+0000 7fa02affd700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 0x7fa0140323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa01c0096a0 tx=0x7fa01c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:16.396 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.394+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa0200095f0 con 0x7fa02c0f5410 2024-02-19T22:25:16.601 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.599+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa02c045ad0 con 0x7fa02c0f5410 2024-02-19T22:25:16.602 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.599+0000 7fa013fff700 1 -- 172.21.15.203:0/240050258 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa02001db70 con 0x7fa02c0f5410 2024-02-19T22:25:16.602 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:16.602 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:16.603 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 msgr2=0x7fa0140323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 0x7fa0140323d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fa01c0096a0 tx=0x7fa01c006b40).stop 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 msgr2=0x7fa02c109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7fa020005010 tx=0x7fa02000b040).stop 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 shutdown_connections 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7fa01402ff10 0x7fa0140323d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 --2- 172.21.15.203:0/240050258 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7fa02c0f5410 0x7fa02c109500 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 >> 172.21.15.203:0/240050258 conn(0x7fa02c0f0b70 msgr2=0x7fa02c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 shutdown_connections 2024-02-19T22:25:16.604 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:16.601+0000 7fa031b6e700 1 -- 172.21.15.203:0/240050258 wait complete. 2024-02-19T22:25:16.605 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:16.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:16 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:14.741491+0000 mgr.smithi111.ftjzov (mgr.14162) 369 : cluster [DBG] pgmap v353: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:17.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:16.600891+0000 mon.smithi111 (mon.0) 333 : audit [DBG] from='client.? 172.21.15.203:0/240050258' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:17.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:17.307622+0000 mon.smithi111 (mon.0) 334 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/mirror_snapshot_schedule"}]: dispatch 2024-02-19T22:25:17.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:17 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:17.308508+0000 mon.smithi111 (mon.0) 335 : audit [INF] from='mgr.14162 172.21.15.111:0/2523995916' entity='mgr.smithi111.ftjzov' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.ftjzov/trash_purge_schedule"}]: dispatch 2024-02-19T22:25:18.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:18.227 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:18.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:18 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:16.741876+0000 mgr.smithi111.ftjzov (mgr.14162) 370 : cluster [DBG] pgmap v354: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.234+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3122748767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 msgr2=0x7f1b480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.234+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3122748767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b480f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f1b3c009fb0 tx=0x7f1b3c009b30).stop 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.234+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3122748767 shutdown_connections 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.234+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3122748767 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b480f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.234+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3122748767 >> 172.21.15.203:0/3122748767 conn(0x7f1b480f1c70 msgr2=0x7f1b480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3122748767 shutdown_connections 2024-02-19T22:25:20.239 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3122748767 wait complete. 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 Processor -- start 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 -- start start 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4f2db700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b48106cd0 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4d077700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4d077700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46110/0 (socket says 172.21.15.203:46110) 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.235+0000 7f1b4d077700 1 -- 172.21.15.203:0/3580122664 learned_addr learned my addr 172.21.15.203:0/3580122664 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b4d077700 1 -- 172.21.15.203:0/3580122664 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c005040 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b4d077700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1b3c005010 tx=0x7f1b3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b3c009320 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b3c013380 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b48106ed0 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b3c014e50 con 0x7f1b480f6510 2024-02-19T22:25:20.240 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.236+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b48107360 con 0x7f1b480f6510 2024-02-19T22:25:20.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.237+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b48045ad0 con 0x7f1b480f6510 2024-02-19T22:25:20.241 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.237+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b3c0095f0 con 0x7f1b480f6510 2024-02-19T22:25:20.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.237+0000 7f1b39ffb700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 0x7f1b340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:20.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.237+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b3c040ef0 con 0x7f1b480f6510 2024-02-19T22:25:20.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.237+0000 7f1b4c876700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 0x7f1b340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:20.242 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.238+0000 7f1b4c876700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 0x7f1b340323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f1b440096a0 tx=0x7f1b44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:20.244 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.242+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b3c014420 con 0x7f1b480f6510 2024-02-19T22:25:20.451 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.448+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b481081b0 con 0x7f1b480f6510 2024-02-19T22:25:20.452 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.449+0000 7f1b39ffb700 1 -- 172.21.15.203:0/3580122664 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b3c010230 con 0x7f1b480f6510 2024-02-19T22:25:20.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:20.452 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 msgr2=0x7f1b340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 0x7f1b340323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f1b440096a0 tx=0x7f1b44006b40).stop 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 msgr2=0x7f1b481067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1b3c005010 tx=0x7f1b3c00b040).stop 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 shutdown_connections 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f1b3402ff10 0x7f1b340323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 --2- 172.21.15.203:0/3580122664 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f1b480f6510 0x7f1b481067e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 >> 172.21.15.203:0/3580122664 conn(0x7f1b480f1c70 msgr2=0x7f1b480f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 shutdown_connections 2024-02-19T22:25:20.454 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:20.451+0000 7f1b4f2db700 1 -- 172.21.15.203:0/3580122664 wait complete. 2024-02-19T22:25:20.455 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:20.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:20 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:18.742183+0000 mgr.smithi111.ftjzov (mgr.14162) 371 : cluster [DBG] pgmap v355: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:21.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:21 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:20.450451+0000 mon.smithi111 (mon.0) 336 : audit [DBG] from='client.? 172.21.15.203:0/3580122664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:22.114 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:22.114 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:22.709 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:22 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:20.742569+0000 mgr.smithi111.ftjzov (mgr.14162) 372 : cluster [DBG] pgmap v356: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:24.119 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 -- 172.21.15.203:0/902944605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 msgr2=0x7ff6b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/902944605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b00f66c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7ff6ac009fb0 tx=0x7ff6ac009b30).stop 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 -- 172.21.15.203:0/902944605 shutdown_connections 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/902944605 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b00f66c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 -- 172.21.15.203:0/902944605 >> 172.21.15.203:0/902944605 conn(0x7ff6b00f1aa0 msgr2=0x7ff6b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 -- 172.21.15.203:0/902944605 shutdown_connections 2024-02-19T22:25:24.120 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 -- 172.21.15.203:0/902944605 wait complete. 2024-02-19T22:25:24.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.118+0000 7ff6b6d30700 1 Processor -- start 2024-02-19T22:25:24.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b6d30700 1 -- start start 2024-02-19T22:25:24.122 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b6d30700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b6d30700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0106ac0 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b5d2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b5d2e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:46126/0 (socket says 172.21.15.203:46126) 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b5d2e700 1 -- 172.21.15.203:0/1303130667 learned_addr learned my addr 172.21.15.203:0/1303130667 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.119+0000 7ff6b5d2e700 1 -- 172.21.15.203:0/1303130667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac005040 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6b5d2e700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7ff6ac0089f0 tx=0x7ff6ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac009320 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b0106cc0 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6ac013380 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6ac014e20 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b0107150 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.120+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6ac014420 con 0x7ff6b00f62a0 2024-02-19T22:25:24.123 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.121+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0045ad0 con 0x7ff6b00f62a0 2024-02-19T22:25:24.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.121+0000 7ff6a67fc700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 0x7ff69c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:24.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.121+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6ac040ff0 con 0x7ff6b00f62a0 2024-02-19T22:25:24.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.121+0000 7ff6b552d700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 0x7ff69c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:24.124 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.121+0000 7ff6b552d700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 0x7ff69c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ff6a00096a0 tx=0x7ff6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:24.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.125+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6ac0095f0 con 0x7ff6b00f62a0 2024-02-19T22:25:24.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.332+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6b0108000 con 0x7ff6b00f62a0 2024-02-19T22:25:24.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.333+0000 7ff6a67fc700 1 -- 172.21.15.203:0/1303130667 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff6ac01db70 con 0x7ff6b00f62a0 2024-02-19T22:25:24.335 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:24.335 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:24.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.334+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 msgr2=0x7ff69c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:24.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.334+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 0x7ff69c0323d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7ff6a00096a0 tx=0x7ff6a0006b40).stop 2024-02-19T22:25:24.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.334+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 msgr2=0x7ff6b01065d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.334+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7ff6ac0089f0 tx=0x7ff6ac00b040).stop 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 shutdown_connections 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7ff69c02ff10 0x7ff69c0323d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 --2- 172.21.15.203:0/1303130667 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7ff6b00f62a0 0x7ff6b01065d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 >> 172.21.15.203:0/1303130667 conn(0x7ff6b00f1aa0 msgr2=0x7ff6b00f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 shutdown_connections 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:24.335+0000 7ff6b6d30700 1 -- 172.21.15.203:0/1303130667 wait complete. 2024-02-19T22:25:24.337 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:24.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:22.742912+0000 mgr.smithi111.ftjzov (mgr.14162) 373 : cluster [DBG] pgmap v357: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:24.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:24 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:24.333264+0000 mon.smithi111 (mon.0) 337 : audit [DBG] from='client.? 172.21.15.203:0/1303130667' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:26.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:26.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:26.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:26 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:24.743182+0000 mgr.smithi111.ftjzov (mgr.14162) 374 : cluster [DBG] pgmap v358: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:28.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- 172.21.15.203:0/3152948494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 msgr2=0x7f83a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:28.044 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 --2- 172.21.15.203:0/3152948494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a80f6930 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f83a4009fb0 tx=0x7f83a4009b30).stop 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- 172.21.15.203:0/3152948494 shutdown_connections 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 --2- 172.21.15.203:0/3152948494 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a80f6930 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- 172.21.15.203:0/3152948494 >> 172.21.15.203:0/3152948494 conn(0x7f83a80f1c70 msgr2=0x7f83a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- 172.21.15.203:0/3152948494 shutdown_connections 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- 172.21.15.203:0/3152948494 wait complete. 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 Processor -- start 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- start start 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.042+0000 7f83afac8700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a8106cd0 con 0x7f83a80f6510 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83ad864700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83ad864700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:57924/0 (socket says 172.21.15.203:57924) 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83ad864700 1 -- 172.21.15.203:0/2972069126 learned_addr learned my addr 172.21.15.203:0/2972069126 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:28.045 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83ad864700 1 -- 172.21.15.203:0/2972069126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a4005040 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83ad864700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f83a4005010 tx=0x7f83a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4009320 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a8106ed0 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f83a4013380 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83a4014e50 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.043+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a8107360 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.044+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83a40095f0 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.044+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a8045ad0 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.044+0000 7f839e7fc700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 0x7f83940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.044+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83a4040fa0 con 0x7f83a80f6510 2024-02-19T22:25:28.047 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.045+0000 7f83ad063700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 0x7f83940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:28.048 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.045+0000 7f83ad063700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 0x7f83940323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f83980096a0 tx=0x7f8398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:28.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.049+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83a4014420 con 0x7f83a80f6510 2024-02-19T22:25:28.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.253+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83a8108210 con 0x7f83a80f6510 2024-02-19T22:25:28.255 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.253+0000 7f839e7fc700 1 -- 172.21.15.203:0/2972069126 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f83a4010230 con 0x7f83a80f6510 2024-02-19T22:25:28.255 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:28.255 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 msgr2=0x7f83940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 0x7f83940323d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f83980096a0 tx=0x7f8398006b40).stop 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 msgr2=0x7f83a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f83a4005010 tx=0x7f83a400b040).stop 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 shutdown_connections 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f839402ff10 0x7f83940323d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 --2- 172.21.15.203:0/2972069126 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f83a80f6510 0x7f83a81067e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 >> 172.21.15.203:0/2972069126 conn(0x7f83a80f1c70 msgr2=0x7f83a80f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 shutdown_connections 2024-02-19T22:25:28.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:28.255+0000 7f83afac8700 1 -- 172.21.15.203:0/2972069126 wait complete. 2024-02-19T22:25:28.258 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:28.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:26.743553+0000 mgr.smithi111.ftjzov (mgr.14162) 375 : cluster [DBG] pgmap v359: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:28.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:28 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:28.253887+0000 mon.smithi111 (mon.0) 338 : audit [DBG] from='client.? 172.21.15.203:0/2972069126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:29.855 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:29.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:30.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:30 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:28.743905+0000 mgr.smithi111.ftjzov (mgr.14162) 376 : cluster [DBG] pgmap v360: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.746+0000 7f34c4502700 1 -- 172.21.15.203:0/2070837064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 msgr2=0x7f34bc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.746+0000 7f34c4502700 1 --2- 172.21.15.203:0/2070837064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc0f90f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f34ac009fb0 tx=0x7f34ac009b30).stop 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- 172.21.15.203:0/2070837064 shutdown_connections 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 --2- 172.21.15.203:0/2070837064 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc0f90f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- 172.21.15.203:0/2070837064 >> 172.21.15.203:0/2070837064 conn(0x7f34bc0f0b70 msgr2=0x7f34bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:31.751 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- 172.21.15.203:0/2070837064 shutdown_connections 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- 172.21.15.203:0/2070837064 wait complete. 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 Processor -- start 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- start start 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c4502700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34bc106c70 con 0x7f34bc060280 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c229e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.747+0000 7f34c229e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:57934/0 (socket says 172.21.15.203:57934) 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34c229e700 1 -- 172.21.15.203:0/556769804 learned_addr learned my addr 172.21.15.203:0/556769804 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34c229e700 1 -- 172.21.15.203:0/556769804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac005040 con 0x7f34bc060280 2024-02-19T22:25:31.752 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34c229e700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f34ac0089f0 tx=0x7f34ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:31.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ac009320 con 0x7f34bc060280 2024-02-19T22:25:31.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f34ac013380 con 0x7f34bc060280 2024-02-19T22:25:31.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34bc106e70 con 0x7f34bc060280 2024-02-19T22:25:31.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34ac014e20 con 0x7f34bc060280 2024-02-19T22:25:31.753 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.748+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34bc107300 con 0x7f34bc060280 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.749+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34ac0095f0 con 0x7f34bc060280 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.749+0000 7f34b2ffd700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 0x7f34a80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.749+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34ac040e90 con 0x7f34bc060280 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.749+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34bc045ad0 con 0x7f34bc060280 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.749+0000 7f34c1a9d700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 0x7f34a80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:31.754 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.750+0000 7f34c1a9d700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 0x7f34a80323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:31.756 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.754+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34ac014420 con 0x7f34bc060280 2024-02-19T22:25:31.962 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.959+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34bc1081b0 con 0x7f34bc060280 2024-02-19T22:25:31.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.960+0000 7f34b2ffd700 1 -- 172.21.15.203:0/556769804 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f34ac01d980 con 0x7f34bc060280 2024-02-19T22:25:31.963 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:31.963 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:31.964 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 msgr2=0x7f34a80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 0x7f34a80323d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f34b80096a0 tx=0x7f34b8006b40).stop 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 msgr2=0x7f34bc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f34ac0089f0 tx=0x7f34ac00b040).stop 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 shutdown_connections 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f34a802ff10 0x7f34a80323d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 --2- 172.21.15.203:0/556769804 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f34bc060280 0x7f34bc106780 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 >> 172.21.15.203:0/556769804 conn(0x7f34bc0f0b70 msgr2=0x7f34bc0f15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 shutdown_connections 2024-02-19T22:25:31.965 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:31.962+0000 7f34c4502700 1 -- 172.21.15.203:0/556769804 wait complete. 2024-02-19T22:25:31.966 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:30.744162+0000 mgr.smithi111.ftjzov (mgr.14162) 377 : cluster [DBG] pgmap v361: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:32.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:32 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:31.961409+0000 mon.smithi111 (mon.0) 339 : audit [DBG] from='client.? 172.21.15.203:0/556769804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:33.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:33.622 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:34.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:34 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:32.744548+0000 mgr.smithi111.ftjzov (mgr.14162) 378 : cluster [DBG] pgmap v362: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:35.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 -- 172.21.15.203:0/2310080017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 msgr2=0x7f82ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:35.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 --2- 172.21.15.203:0/2310080017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac0f6930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f82a0009fb0 tx=0x7f82a0009b30).stop 2024-02-19T22:25:35.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 -- 172.21.15.203:0/2310080017 shutdown_connections 2024-02-19T22:25:35.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 --2- 172.21.15.203:0/2310080017 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac0f6930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:35.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 -- 172.21.15.203:0/2310080017 >> 172.21.15.203:0/2310080017 conn(0x7f82ac0f1c70 msgr2=0x7f82ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:35.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 -- 172.21.15.203:0/2310080017 shutdown_connections 2024-02-19T22:25:35.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.667+0000 7f82b10fb700 1 -- 172.21.15.203:0/2310080017 wait complete. 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82b10fb700 1 Processor -- start 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82b10fb700 1 -- start start 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82b10fb700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82b10fb700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac106cd0 con 0x7f82ac0f6510 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82aad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82aad9d700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39636/0 (socket says 172.21.15.203:39636) 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.668+0000 7f82aad9d700 1 -- 172.21.15.203:0/445630791 learned_addr learned my addr 172.21.15.203:0/445630791 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82aad9d700 1 -- 172.21.15.203:0/445630791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0005040 con 0x7f82ac0f6510 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82aad9d700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f82a0005010 tx=0x7f82a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:35.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a0009320 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac106ed0 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f82a0013380 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82a0014e50 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.669+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac107360 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82a0014420 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac045ad0 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82937fe700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 0x7f82940323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82a0040e30 con 0x7f82ac0f6510 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82aa59c700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 0x7f82940323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:35.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.670+0000 7f82aa59c700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 0x7f82940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:35.677 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.674+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82a00095f0 con 0x7f82ac0f6510 2024-02-19T22:25:35.883 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.880+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82ac108210 con 0x7f82ac0f6510 2024-02-19T22:25:35.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.881+0000 7f82937fe700 1 -- 172.21.15.203:0/445630791 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f82a001db70 con 0x7f82ac0f6510 2024-02-19T22:25:35.884 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:35.884 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:35.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 msgr2=0x7f82940323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:35.886 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 0x7f82940323d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f829c0096a0 tx=0x7f829c006b40).stop 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 msgr2=0x7f82ac1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f82a0005010 tx=0x7f82a000b040).stop 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 shutdown_connections 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f829402ff10 0x7f82940323d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 --2- 172.21.15.203:0/445630791 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f82ac0f6510 0x7f82ac1067e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:35.887 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 >> 172.21.15.203:0/445630791 conn(0x7f82ac0f1c70 msgr2=0x7f82ac0f26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:35.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 shutdown_connections 2024-02-19T22:25:35.888 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:35.883+0000 7f82b10fb700 1 -- 172.21.15.203:0/445630791 wait complete. 2024-02-19T22:25:35.888 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:34.744915+0000 mgr.smithi111.ftjzov (mgr.14162) 379 : cluster [DBG] pgmap v363: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:36.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:36 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:35.882250+0000 mon.smithi111 (mon.0) 340 : audit [DBG] from='client.? 172.21.15.203:0/445630791' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:37.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T22:25:37.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T22:25:38.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:38 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:36.745312+0000 mgr.smithi111.ftjzov (mgr.14162) 380 : cluster [DBG] pgmap v364: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:39.605 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 -- 172.21.15.203:0/4048840874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 msgr2=0x7f02200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 --2- 172.21.15.203:0/4048840874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02200f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f0210009fb0 tx=0x7f0210009b30).stop 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 -- 172.21.15.203:0/4048840874 shutdown_connections 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 --2- 172.21.15.203:0/4048840874 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02200f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 -- 172.21.15.203:0/4048840874 >> 172.21.15.203:0/4048840874 conn(0x7f02200f0b70 msgr2=0x7f02200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 -- 172.21.15.203:0/4048840874 shutdown_connections 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.602+0000 7f022843e700 1 -- 172.21.15.203:0/4048840874 wait complete. 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f022843e700 1 Processor -- start 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f022843e700 1 -- start start 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f022843e700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:39.606 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f022843e700 1 -- --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0220108ec0 con 0x7f02200f5410 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f02261da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f02261da700 1 --2- >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.111:3300/0 says I am v2:172.21.15.203:39668/0 (socket says 172.21.15.203:39668) 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.603+0000 7f02261da700 1 -- 172.21.15.203:0/151069799 learned_addr learned my addr 172.21.15.203:0/151069799 (peer_addr_for_me v2:172.21.15.203:0/0) 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f02261da700 1 -- 172.21.15.203:0/151069799 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210005040 con 0x7f02200f5410 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f02261da700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0210005010 tx=0x7f021000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0210009320 con 0x7f02200f5410 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02201090c0 con 0x7f02200f5410 2024-02-19T22:25:39.609 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 2 ==== config(26 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0210013380 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0210014e50 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.604+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0220109550 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.605+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0210014420 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.605+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0220045ad0 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.605+0000 7f0216ffd700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 0x7f020c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.605+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0210040e30 con 0x7f02200f5410 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.605+0000 7f02259d9700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 0x7f020c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-02-19T22:25:39.610 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.606+0000 7f02259d9700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 0x7f020c0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f021c0096a0 tx=0x7f021c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T22:25:39.612 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.610+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02100095f0 con 0x7f02200f5410 2024-02-19T22:25:39.818 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.815+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 --> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f022010a400 con 0x7f02200f5410 2024-02-19T22:25:39.819 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.816+0000 7f0216ffd700 1 -- 172.21.15.203:0/151069799 <== mon.0 v2:172.21.15.111:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f021001db70 con 0x7f02200f5410 2024-02-19T22:25:39.819 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-02-19T22:25:39.819 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":1,"fsid":"dbbe4146-cf73-11ee-95bb-87774f69a715","modified":"2024-02-19T22:12:08.292237Z","created":"2024-02-19T22:12:08.292237Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T22:25:39.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 msgr2=0x7f020c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 0x7f020c0323d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f021c0096a0 tx=0x7f021c006b40).stop 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 msgr2=0x7f02201089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f0210005010 tx=0x7f021000b040).stop 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 shutdown_connections 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:6800/1850216637,v1:172.21.15.111:6801/1850216637] conn(0x7f020c02ff10 0x7f020c0323d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 --2- 172.21.15.203:0/151069799 >> [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] conn(0x7f02200f5410 0x7f02201089d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 >> 172.21.15.203:0/151069799 conn(0x7f02200f0b70 msgr2=0x7f022005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 shutdown_connections 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-02-19T22:25:39.818+0000 7f022843e700 1 -- 172.21.15.203:0/151069799 wait complete. 2024-02-19T22:25:39.821 INFO:teuthology.orchestra.run.smithi203.stderr:dumped monmap epoch 1 2024-02-19T22:25:40.473 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-02-19T22:25:40.474 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T22:25:40.474 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:25:40.498 DEBUG:teuthology.orchestra.run.smithi203:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T22:25:40.516 INFO:tasks.cephadm:Stopping all daemons... 2024-02-19T22:25:40.516 INFO:tasks.cephadm.mon.smithi111:Stopping mon.smithi111... 2024-02-19T22:25:40.516 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111 2024-02-19T22:25:40.706 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: cluster 2024-02-19T22:25:38.745741+0000 mgr.smithi111.ftjzov (mgr.14162) 381 : cluster [DBG] pgmap v365: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T22:25:40.706 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: audit 2024-02-19T22:25:39.817415+0000 mon.smithi111 (mon.0) 341 : audit [DBG] from='client.? 172.21.15.203:0/151069799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T22:25:40.707 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:40 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for dbbe4146-cf73-11ee-95bb-87774f69a715... 2024-02-19T22:25:40.707 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:25:40.702+0000 7fb6d61c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-02-19T22:25:40.960 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:40 smithi111 ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111[38195]: debug 2024-02-19T22:25:40.702+0000 7fb6d61c1700 -1 mon.smithi111@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T22:25:41.460 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Feb 19 22:25:41 smithi111 podman[44090]: ceph-dbbe4146-cf73-11ee-95bb-87774f69a715-mon.smithi111 2024-02-19T22:25:41.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi111.service' 2024-02-19T22:25:41.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T22:25:41.614 INFO:tasks.cephadm.mon.smithi111:Stopped mon.smithi111 2024-02-19T22:25:41.614 INFO:tasks.cephadm.mon.smithi203:Stopping mon.smithi203... 2024-02-19T22:25:41.614 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203 2024-02-19T22:25:41.653 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203.service: Unit ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203.service not loaded. 2024-02-19T22:25:41.654 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-02-19T22:25:41.654 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi203" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 5: 'sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203' 2024-02-19T22:25:41.655 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T22:25:41.657 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/ubuntu@smithi111.front.sepia.ceph.com/crash 2024-02-19T22:25:41.658 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:25:41.714 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/ubuntu@smithi203.front.sepia.ceph.com/crash 2024-02-19T22:25:41.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/crash -- . 2024-02-19T22:25:41.763 INFO:teuthology.orchestra.run.smithi203.stderr:tar: /var/lib/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/crash: Cannot open: No such file or directory 2024-02-19T22:25:41.764 INFO:teuthology.orchestra.run.smithi203.stderr:tar: Error is not recoverable: exiting now 2024-02-19T22:25:41.764 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T22:25:41.765 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v 'Degraded data redundancy' | egrep -v '\(PG_' | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v 'deprecated feature inline_data' | egrep -v 'compat changed unexpectedly' | egrep -v 'overall HEALTH_' | egrep -v OSD_DOWN | egrep -v OSD_ | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T22:25:41.801 INFO:tasks.cephadm:Compressing logs... 2024-02-19T22:25:41.801 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:25:41.844 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T22:25:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2024-02-19T22:25:41.870 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:25:41.870 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi111.log 2024-02-19T22:25:41.871 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi111.log: gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log: /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi111.ftjzov.log 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr: 88.5% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log: /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi111.ftjzov.log: 90.3% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr: 88.9% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T22:25:41.872 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.log 2024-02-19T22:25:41.873 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.log: 88.7% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T22:25:41.874 INFO:teuthology.orchestra.run.smithi203.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-19T22:25:41.874 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T22:25:41.874 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T22:25:41.875 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log: 88.4% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T22:25:41.876 INFO:teuthology.orchestra.run.smithi111.stderr: 91.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:25:41.876 INFO:teuthology.orchestra.run.smithi203.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T22:25:41.877 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-02-19T22:25:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.015s 2024-02-19T22:25:41.877 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m0.012s 2024-02-19T22:25:41.878 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.013s 2024-02-19T22:25:41.968 INFO:teuthology.orchestra.run.smithi111.stderr: 92.6% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mgr.smithi111.ftjzov.log.gz 2024-02-19T22:25:42.043 INFO:teuthology.orchestra.run.smithi111.stderr: 90.6% -- replaced with /var/log/ceph/dbbe4146-cf73-11ee-95bb-87774f69a715/ceph-mon.smithi111.log.gz 2024-02-19T22:25:42.044 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-02-19T22:25:42.044 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m0.185s 2024-02-19T22:25:42.044 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m0.273s 2024-02-19T22:25:42.044 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.026s 2024-02-19T22:25:42.045 INFO:tasks.cephadm:Archiving logs... 2024-02-19T22:25:42.045 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/ubuntu@smithi111.front.sepia.ceph.com/log 2024-02-19T22:25:42.046 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:25:42.168 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/ubuntu@smithi203.front.sepia.ceph.com/log 2024-02-19T22:25:42.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-19T22:25:42.202 INFO:tasks.cephadm:Removing cluster... 2024-02-19T22:25:42.202 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:28:27.721 DEBUG:teuthology.orchestra.run.smithi203:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid dbbe4146-cf73-11ee-95bb-87774f69a715 --force 2024-02-19T22:28:28.023 INFO:tasks.cephadm:Removing cephadm ... 2024-02-19T22:28:28.023 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:28:28.044 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-19T22:28:28.064 INFO:tasks.cephadm:Teardown complete 2024-02-19T22:28:28.065 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 5: 'sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203' 2024-02-19T22:28:28.320 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=777c7a7cc6c843138ac8a1d9271dd5b0 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_81bd20d634209c7cb82c18be12b4b5a05643ebf1/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 5: 'sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203' 2024-02-19T22:28:28.325 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T22:28:28.334 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T22:28:28.350 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T22:28:28.350 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:28:28.380 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T22:28:28.478 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T22:28:28.490 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T22:28:28.491 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:28:28.493 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T22:28:28.507 INFO:teuthology.orchestra.run.smithi111.stderr:bash: ntpq: command not found 2024-02-19T22:28:28.511 INFO:teuthology.orchestra.run.smithi203.stderr:bash: ntpq: command not found 2024-02-19T22:28:28.512 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:28:28.513 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-02-19T22:28:28.513 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1946us[+1951us] +/- 107ms 2024-02-19T22:28:28.513 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1605us[-1600us] +/- 59ms 2024-02-19T22:28:28.513 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +1002us[+1007us] +/- 93ms 2024-02-19T22:28:28.513 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:28:28.514 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T22:28:28.514 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-02-19T22:28:28.514 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1945us[+1945us] +/- 107ms 2024-02-19T22:28:28.515 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1589us[-1590us] +/- 59ms 2024-02-19T22:28:28.515 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +1015us[+1015us] +/- 93ms 2024-02-19T22:28:28.515 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T22:28:28.515 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T22:28:28.526 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T22:28:28.526 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T22:28:28.536 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:28:28.612 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T22:28:28.677 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:28:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:28:28.707 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T22:28:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:28:28.737 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T22:28:28.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T22:28:28.756 INFO:teuthology.task.internal:Duration was 1989.778354 seconds 2024-02-19T22:28:28.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T22:28:28.766 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T22:28:28.766 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:28:28.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T22:28:28.812 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:28:28.823 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T22:28:29.094 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-19T22:28:29.094 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-02-19T22:28:29.095 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:28:29.118 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-02-19T22:28:29.118 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-19T22:28:29.140 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-19T22:28:29.141 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:28:29.162 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-19T22:28:29.230 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-19T22:28:29.230 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:28:29.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-19T22:28:29.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T22:28:29.332 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T22:28:29.333 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:28:29.360 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T22:28:29.387 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T22:28:29.398 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:28:29.402 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:28:29.426 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-02-19T22:28:29.453 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-02-19T22:28:29.488 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:28:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:28:29.505 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-19T22:28:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T22:28:29.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T22:28:29.556 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T22:28:29.557 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/smithi111 2024-02-19T22:28:29.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:28:29.598 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796/remote/smithi203 2024-02-19T22:28:29.599 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T22:28:29.638 INFO:teuthology.task.internal:Removing archive directory... 2024-02-19T22:28:29.638 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:28:29.641 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-19T22:28:29.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T22:28:29.703 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T22:28:29.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T22:28:29.713 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T22:28:29.713 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:28:29.715 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T22:28:29.730 INFO:teuthology.orchestra.run.smithi111.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:28 /home/ubuntu/cephtest 2024-02-19T22:28:29.747 INFO:teuthology.orchestra.run.smithi203.stdout: 393225 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 22:28 /home/ubuntu/cephtest 2024-02-19T22:28:29.749 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T22:28:29.761 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T22:28:29.785 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T22:28:29.815 DEBUG:teuthology.nuke:shortname: smithi111 2024-02-19T22:28:29.816 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:28:29.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:02.177237', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJXcUc/9/OJrMEBUbnTBFhGVF+al70FV2WM6tLlZX8j5d9oYSE+bcqUD9msMfgrZwmtQddO0HbJsYCjMn3tO1Q8='} 2024-02-19T22:28:29.872 DEBUG:teuthology.nuke:shortname: smithi203 2024-02-19T22:28:29.873 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T22:28:29.908 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-19_19:25:49-rados-pacific-release-distro-default-smithi/7566796', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 21:46:02.179344', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFrkOGisSEJpUGGy6TtGBDenHntsMT3eWmS78WfWD0/XFkXwbIom7RWNLVikoSzy9juQ+rAqsmg2gFTfInoqAXs='} 2024-02-19T22:28:29.963 INFO:teuthology.orchestra.console.smithi111:Power off 2024-02-19T22:28:29.963 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:28:30.015 INFO:teuthology.orchestra.console.smithi203:Power off 2024-02-19T22:28:30.015 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T22:28:30.027 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-02-19T22:28:30.027 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:30.068 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-02-19T22:28:30.068 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:30.079 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-02-19T22:28:30.103 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-02-19T22:28:34.080 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:34.104 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:34.196 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-02-19T22:28:34.215 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-02-19T22:28:38.200 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:38.216 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:38.312 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-02-19T22:28:38.326 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-02-19T22:28:38.426 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-02-19T22:28:42.316 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T22:28:42.427 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-02-19T22:28:42.527 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-02-19T22:28:42.632 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1989.7783544063568 failure_reason: 'Command failed on smithi203 with status 5: ''sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=777c7a7cc6c843138ac8a1d9271dd5b0 status: fail success: false 2024-02-19T22:28:42.632 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T22:28:42.731 INFO:teuthology.run:FAIL